Unit 1.5: Global and Local Optima

A major challenge in optimization is distinguishing between the "best in the neighborhood" and the "best in the world."

1. Local Minimum

A point X is a relative or local minimum if f(X) <= f(X) for all X in the immediate neighborhood (small distance δ) of X*.

• Analogy: You are standing at the bottom of a small pit. Every step you take in any direction leads up. You are at a local minimum.

2. Global Minimum

A point X is a global minimum if f(X) <= f(X) for all feasible X in the entire design space.

• Analogy: The deepest point of the Dead Sea. No other point on Earth is lower.

3. Visualization ("The Multimodal Function")

Imagine a function curve that looks like a "W" shape.

• The curve goes down, hits a bottom (Point A), goes up a bit, goes down again to a deeper bottom (Point B), and goes up again.
• Point A: Local Minimum (It's low, but not the lowest).
• Point B: Global Minimum (The absolute lowest point).
• Optimization Trap: Simple algorithms (like Gradient Descent) often get "stuck" in Local Minima (Point A) because they only look downhill locally.

4. Convexity

• Convex Function: A bowl-shaped function (like y = x²). It has only one minimum, which is automatically the Global Minimum. This is the ideal case.
• Non-Convex Function: A wavy function (like y = x² + 10sin(x)). Has multiple peaks and valleys. Harder to solve.