Lecture 9 - Direct Methods Part 1 Flashcards
Define Direct Methods
A class of optimisation algorithms that use only objective function evaluations to find optima, without using gradient information.
- Direct Methods rely solely on the objective function (cost function, loss function, fitness,…)
- Each time you evaluate a candidate solution, you’re not just getting a number for the candidate you learn a bit more about the space
What are the characteristics of Direct Methods?
- Derivatives are not available
○ Search space may be smooth, but very complex
○ Search space may not be differentiable
○ Search space may not be continuous
What is the worst case for Direct Methods?
Worst case the search space might be a black box and at the very least we might be able to evaluate it
Black Box Diagram and Explanation
REFER TO SLIDES
What is the purpose of the Black Box?
§ Try a solution
§ Evaluate its performance
§ Adjust parameters
§ Repeat
○ Continue this loop until a satisfactory or optimal solution is found.
What is a Cyclic Coordinate Search?
A type of direct search where one variable (coordinate) is optimised at a time in a cyclic order.
What is the procedure of Cyclic Coordinate Search?
Procedure:
1. Start at an initial point x⃗0.
2. Perform a line search along x1, then fix x1 and search along x2, etc.
3. After all coordinates are updated, repeat the cycle.
What are the advantages and limitations of Cyclic Coordinate Search?
Advantages:
- Simple to implement.
- Guaranteed to not worsen the objective value with each step.
Limitations:
- Can miss local optima.
- Slower convergence in narrow “valleys” aligned diagonally.
What is the Stopping Criteria for Cyclic Coordinate Search?
When the change in the vector is small enough -> L2 Norm
REFER TO NOTES
What is a Line Search?
Line Search is a tool, not a full optimisation method.
- It’s a subroutine used in many optimisation algorithms — including CCS.
- In Cyclic Coordinate Search, once you fix all variables except one (e.g., fix y and vary x), you perform a line search along that axis to find the value of x that minimizes the function.
What is the purpose of a Line Search?
To find the minimum of a function along a single direction (line) — without using derivatives.
This is essential in direct methods, where gradients are unknown or unavailable.
What are the common Line Search Techniques?
Golden Section Search
Fibonacci Search
What is the Golden Section Search?
Uses a golden ratio -> REFER TO NOTES
○ Efficiently narrows the interval [a,b] in which the minimum lies
○ Requires only function evaluations, not derivatives
How does the Golden Selection Search work?
- How it works:
- Choose interval [a,b]
- Compute two interior points using the golden ratio
- Evaluate the function at those points
- Keep the subinterval where the minimum lies
- Repeat until the interval is small enough
REFER TO NOTES FOR EXAMPLE
What are the advantages of the Golden Selection Search
- Advantages:
○ Very efficient
○ Simple and deterministic
○ Good balance between speed and accuracy
○ Works with any unimodal function
What is the Fibonacci Search?
Similar to Golden Section, but it uses the Fibonacci sequence to determine how to divide the interval
How does the Fibonacci Search work?
○ Suppose you want to do n evaluations
○ Pre-compute Fn, the nth Fibonacci number
○ Use ratios of Fibonacci numbers to divide the interval in each step
REFER TO NOTES FOR EXAMPLE
What are the advantages of the Fibonacci Search?
○ Guarantees convergence in a fixed number of steps
○ Minimizes the worst-case number of function evaluations
Describe how line search methods integrate with CCS
In CCS, the optimization proceeds by minimizing the objective function one coordinate at a time. During each coordinate update, a line search method (such as Golden Section Search or Fibonacci Search) is applied to efficiently find the minimum along that axis. These methods do not require gradient information, making them ideal for direct methods like CCS.
Compare the golden section and Fibonacci methods for line search. Why are they suitable for direct methods?
Both the Golden Section Search and the Fibonacci Search are efficient techniques used to minimize a function along a line without requiring derivative information, making them ideal for direct methods.
Golden Section Search uses the golden ratio to iteratively reduce the interval of uncertainty, while Fibonacci Search uses the Fibonacci sequence to determine where to evaluate the function. Fibonacci Search ensures convergence in a predetermined number of evaluations, which can be advantageous when function evaluations are costly. Both methods work well for unimodal functions and are integral to subroutines like those found in Cyclic Coordinate Search (CCS).
What is Cyclic Coordinate Search with “Acceleration Step” - Why add the Acceleration Step?
Why Add an Acceleration Step?
- Problem with Standard CCS:
○ Cyclic Coordinate Search (CCS) optimizes one variable at a time.
○ So it only moves along the coordinate axes (e.g., x-axis, then y-axis, etc.).
○ This works fine in many cases, but…
- In diagonal or curved valleys:
○ The optimal path lies diagonally, not along individual axes.
○ CCS takes a zigzagging path, which is slow and inefficient.
Why should you use the Acceleration Step?
Acceleration helps move towards the true direction of descent, not just along the axes
Speeds up convergence
Describe the acceleration step in Cyclic Coordinate Search and explain how it helps improve convergence.
The acceleration step in Cyclic Coordinate Search is a technique used to improve convergence by addressing the inefficiency of axis-aligned movement. After completing one full cycle of coordinate updates, we compute the net progress vector. This vector represents the overall direction of improvement. An additional step is taken in this direction, helping the algorithm move more directly toward the minimum, especially in diagonally aligned or curved valleys. This strategy accelerates convergence by reducing zigzagging and better approximating the optimal path.
What is Powell’s Method?
An extension of CCS that uses updated search directions rather than just the coordinate axes.
How does Powell’s Method work?
- Step-by-step Breakdown:
- Start with a set of initial directions, usually the standard basis vectors (e.g., e⃗1,e⃗2 for 2D).
- Perform line searches along each direction.
- After all directions are used (a full cycle), calculate the net movement vector:
i. REFER TO SLIDES FOR FORMULA - Perform a line search along this new direction d⃗.
- Update the direction set:
§ Replace the direction that contributed the least to the minimization (often the first one used).
§ Add the new direction d⃗\vec{d}d to the set. - Repeat the cycle using the new updated direction set.
What are the advantages of Powell’s Method?
- Searches in more informative directions than the coordinate axes alone.
- Faster convergence in non-axis-aligned landscapes.
Why is Powell’s Method better than Cyclic Coordinate Search
Adaptive and faster
What improvement does Powell’s Method offer over standard CCS? Why does maintaining a direction queue help?
Powell’s Method improves on standard CCS by dynamically updating its set of search directions rather than relying solely on fixed coordinate axes. After each full cycle of direction-based minimization, it adds a new direction based on the overall progress vector and replaces a less useful direction from the queue. This allows it to explore more informative paths through the search space, particularly in non-axis-aligned landscapes, leading to faster convergence. The direction queue helps by building a basis that better spans the optimization space, even without using gradient information.
What are the linear dependence issues in Powell’s Method, and why is spanning the space important?
In Powell’s Method, the set of search directions is updated over time, which can lead to linear dependence—where one direction becomes a combination of others. This weakens the set’s ability to span the full search space, meaning some directions may no longer be explored. Spanning the space is critical for ensuring the algorithm can reach any point in the domain. If the direction set does not span the space, Powell’s Method may fail to converge or may get trapped in a subspace, missing the global or local minimum.
