Unit 2.3: Newton’s Method

Newton’s method is a "Second-Order" method. It tries to fix the slow convergence of Steepest Descent by using curvature information.

1. Concept

Steepest Descent approximates the function as a flat plane (linear). Newton's method approximates the function as a Quadratic Surface (bowl shape) using the second derivative (Hessian Matrix).

• If the actual function is a perfect quadratic (e.g., f(x) = x²), Newton's method finds the minimum in ONE step.

2. Mathematical Formulation

Search Direction (Sᵢ): Instead of just relying on the gradient, we modify it using the inverse of the Hessian Matrix [J]. Sᵢ = -[J]⁻¹ * ∇f(Xᵢ)

Update Formula: X₍ᵢ₊₁₎ = Xᵢ - [J]⁻¹ * ∇f(Xᵢ)

Where:

• ∇f: Gradient Vector (First derivatives).
• [J]: Hessian Matrix (Second partial derivatives).

3. Algorithm

1. Start: Select initial guess X₁.
2. Compute: Gradient ∇f and Hessian [J] at Xᵢ.
3. Invert: Compute the inverse matrix [J]⁻¹.
4. Update: X₍ᵢ₊₁₎ = Xᵢ - [J]⁻¹ * ∇f.
5. Check Convergence: If gradient is near zero, stop.

4. Pros & Cons