Unit 2.2: Cauchy’s Steepest Descent Method

This is the oldest and simplest gradient-based method, proposed by Cauchy in 1847.

1. Concept

At any point X, the function value decreases most rapidly in the direction of the negative gradient.

• Search Direction: Sᵢ = -∇f(Xᵢ)
• It is a "First-Order" method because it only uses the first derivative.

2. Algorithm (Step-by-Step)

Step 1: Start Select an initial guess X₁ and a convergence tolerance ε (e.g., 0.001). Set iteration counter i = 1.

Step 2: Calculate Gradient Compute the gradient vector at the current point: ∇f(Xᵢ).

Step 3: Check Convergence Calculate the magnitude (norm) of the gradient: |∇f(Xᵢ)|.

• If |∇f(Xᵢ)| < ε, STOP. Xᵢ is the optimum solution.
• Else, continue.

Step 4: Determine Direction Set search direction Sᵢ = -∇f(Xᵢ).

Step 5: Determine Step Size (λᵢ) Find λᵢ such that it minimizes the function f(Xᵢ + λ*Sᵢ).

• This becomes a single-variable optimization problem: Minimize F(λ).

Step 6: Update Calculate new point: X₍ᵢ₊₁₎ = Xᵢ + λᵢ * Sᵢ.

Step 7: Loop Set i = i + 1 and go to Step 2.

3. Pros & Cons