Lecture 9 - Univariate Optimisation

Question 1

What is optimisation?

Answer 1

Finding the maximum or minimum of a function

Question 2

What is the objective function for least squares?

Answer 2

The sum of the residuals squared

Question 3

How to find maximum (log)likelihood?

Answer 3

Propose a model f(x, θ) for the data.
Form a likelihood - L(θ) = the product of f(xi, θ) (OBJECTIVE FUNCTION)
Find the parameter values which maximise the likelihood

Question 4

What is a systematic line search?

Answer 4

Start by bracketing the maximum (find points [a, b] which are known to contain the maximum)
Divide this interval into a regular set of values x, spaced e apart.
Evaluate function at each x
Choose x that has max g(x)

Question 5

What are pros and cons of iterative methods?

Answer 5

Pro:
Can be considerably more efficient

Con:
Can be less reliable

Question 6

What is the general iterative algorithm?

Answer 6

• Start at iteration t=0 begin with an intial guess x(0)
• Start next iteration, t=t+1
• At iteration t, produce an improved guess, x(t) using an updating equation, x(t) = f(x(t-1))
• Evaluate whether new guess is sufficiently accurate using stopping rule
• If yes, stop and return x* = x(t) and g(x*) = g(x(t))
• If no consider whether to report no convergence, otherwise go to back to step 2.

Question 7

What is the algorithm for the bisection method?

Answer 7

• Start an iteration t=0 with (a(0), b(0))
• Set starting value x(0) = (a(0) + b(0))/2
• Start next iteration t=t+1
• Update (a(t), b(t), x(t)) according to:
• If stopping rule met stop and return x* = x(t)
  else go to 3.

Question 8

What is the bisection method?

Answer 8

Simple example of an iterative method for finding the root g’(x) =0

Question 9

What is implied if g’(x) is continuous on some interval [a,b] and g’(a)g’(b) <= 0

Answer 9

There is at least one x* in [a,b] for which g’(x) = 0 making g(x) a local optimum.

Question 10

What is convergence order?

Answer 10

Index of how fast convergence happens

Question 11

What stopping criteria are not reliable?

Answer 11

Criteria based on how close g’(x) is to 0

Question 12

What are the two main types of convergence criteria?

Answer 12

Absolute and relative

Question 13

What is absolute convergence criteria?

Answer 13

Stop when |x(t) - x(t-1)| < e, for some desired maximum imprecision e.

Question 14

What is the problem with absolute convergence criteria?

Answer 14

There is always the same absolute imprecision regardless of the size of numbers.

Question 15

What is convergence criterion?

Answer 15

( |x(t) - x(t-1)| / |x(t-1)| ) < e

Question 16

What are partial remedies for numerical computer imprecision preventing convergence?

Answer 16

• More numerically stable updates
• Good enough convergence
• Stop after n iterations whether convergence achieved or not
• Stop if abs/relative convergence criteria or |g’(x)| fail to decrease over several iterations

Question 17

Why might you miss global optimum?

Answer 17

May have failed to bracket it or there may have been multiple optima and the one bracketed is not the best one.