Lecture 3

Question 1

Solution Methods for Unconstrained Optimization

Answer 1

Indirect methods (Optimal solution is found by solving the system of equations)
and
Direct methods (Optimal solution is found by directly improving the objective function via iterative descent)

Question 2

Direct methods
Iterative descent

Answer 2

Line search approach
and
Trust region approach

Question 3

Armijo Condition

Answer 3

Let
𝑓 be continuously differentiable, 𝒑 a descent direction, and let 𝑐 ∈(0,1) be given. Then there exists an
𝛼>0, such that for 𝜙(𝛼)≔𝑓(𝒙𝑘+𝛼𝒑), the condition 𝜙(𝛼)≤𝜙(0)+𝛼𝑐𝜙′(0) holds

Question 4

Wolfe Conditions

Answer 4

Relevance:
Wolfe Conditions promote
convergence to a stationary point

Geometric interpretation:
guarantee minimum step length!

Question 5

Line search approach

Determination of a Descent Direction

Answer 5

Steepest descent direction
Conjugate directions
Newton’s step direction
…. special cases

Question 6

Zeroth order oracle
First order oracle
Second order oracle

Answer 6

• Given 𝒙return the value 𝑓(𝒙)
• Given 𝒙return 𝑓𝒙and gradient 𝜵𝑓(𝒙)
• Given 𝒙return 𝑓𝒙, 𝜵𝑓(𝒙) and Hessian 𝜵^2𝑓(𝒙)

Question 7

Analytical Complexity
Arithmetical Complexity

Answer 7

Analytical Complexity: The smallest number of queries to an oracle to solve Problem 𝑃 to accuracy ε

Arithmetical Complexity: The smallest number of arithmetic operations (including work of the oracle and work of method), required to solve problem 𝑃up to accuracy ε