Lecture 3 - Optimisation and Hypothesis Space Flashcards
What are the three key ingredients in optimisation?
Three Key ingredients:
1. Representation: A representation of the problem we are trying to solve and therefore we need a language (a way of describing the world)
a. Defines a solution space
2. Instantiation: Potential Candidate Solutions also known as a model
a. Defines candidate solution
3. Evaluation: Emulating the solutions using a metric
a. How good is the model
What are some examples of language (representation)? - As part of the 3 key ingredients
‣ mathematical equations
‣ natural languages
‣ grammars
‣ logics
‣ finite automata/finite-state machines
‣ computer programs
‣ logic programs
‣ Gantt charts
‣ PERT charts
‣ simulation languages
‣ popsticks and glue
As part of language what is common phrase used to determine if something can be modelled?
if you can’t describe it - you can’t model it!
Example: Enormous amount of work on grammars to describe (explain/ parse/generate/model) natural languages
REFER TO EXAMPLES ON SLIDES
What is the difference between tests and generation?
We don’t have to generate the whole space to answer these questions!
* Test
‣ we have a test (in this case called parsing) for whether a construct is valid (in the set of sentences)
‣ finite time
===
* Generate
‣ we can also generate members of the set, but…
‣ infinite time
Usually testing is much cheaper (eg “polynomial time”) than generating (we’ll return to this)!
What do we mean by the expressiveness of a language?
if you can’t describe it - you can’t model it!
We will say that language A is more expressive (or “richer”) than language B if
‣ everything that can be described in language B can also be described in language A (i.e. A subsumes B)
‣ something can be described in language A that cannot be described in language B (B does not subsume A)
What is Model (Instantiation)? - As part of the 3 key ingredients
‣ something that can be described in the selected language
‣ an instance of all the possible things that can be described in the language
‣ may be executable, parameterised, generative, or canonical form
Why is a model considered a hypothesis?
Relationship to “real” world
‣ an approximation to the real world (process, function, structure,…)
‣ an abstraction of the real world
‣ an hypothesis that attempts to describe how the real world works (or could work)
‣ a candidate solution (in the context of optimisation)
===
therefore a model/hypothesis is an attempt to describe/explain some function, within the chosen language
What is a Metric?- As part of the 3 key ingredients
How good is your hypthesis? -> how much is it like the real thing and how close is it to the target
What other names represent metrics and evaluation?
‣ error function/metric
‣ cost function
‣ fitness function
‣ objective function
‣ penalty function
‣ utility function
* (Usually) function from hypothesis space onto
‣ (sometimes we have multiple objectives) = f : H -> ℝ
What are features?
- Just as the choice of language determines what hypotheses are available to choose from, it also impacts what features are available to compare
What are Numerical Features and Errors
‣ contribution to error is distance (difference) between model and sample at each x value
Expressed as a function -> REFER TO SLIDES
What is the first defintion for optimisation?
‣ Find a model
‣ within the hypothesis space (determined by the language)
‣ that is indistinguishable from the target (zero error with respect to the selected features)
* Note 1: if the evaluated features are a “subset” of the description there could be many solutions - equivalent wrt the metric
* Note 2: in practice in complex problems its more likely there is no matching solution within the hypothesis space
REFER TO SLIDES FOR DIAGRAMS
What is the second defintion for optimisation?
Find a model
‣ within the hypothesis space (determined by the language)
‣ that is closest (minimal error) to the target
* Note 1: in complex problems it may be too expensive, or impossible (infinite domains) to find the best (closest) solution
* Note 2: sometimes the target may not be well defined (eg “infinite fitness”)
REFER TO SLIDES FOR DIAGRAMS
What is the practical definition for optimisation?
‣ Find a model
‣ within the hypothesis space (determined by the language)
‣ that is as close as possible (good enough) to the target
‣ within a specified amount of compute (time and space)
* Distinguish between
- online - compute in real time
- offline - better result outweighs time cost
REFER TO SLIDES FOR DIAGRAMS
