Siksha Sarovar

Siksha Sarovar (sikshasarovar.com) is a free educational web application that helps students in India learn programming and prepare for academic and competitive exams. The platform offers structured coding courses (C, C++, Python, Java, HTML, CSS, PHP, Power BI, AI, Machine Learning, Data Science), complete university curriculum notes for BCA/MCA students with previous year question papers, Class 10 and Class 12 CBSE/HBSE school notes, and dedicated preparation material for SSC, UPSC, Banking, Railway and other government exams. Browsing the site is completely free and requires no account. Users may optionally sign in with Google solely to save their learning progress, quiz scores and personal preferences across devices.

Privacy Policy | Terms of Service | Contact Siksha Sarovar | About Siksha Sarovar

v4.0.9 · PWA
Siksha Sarovar logo
Siksha Sarovar
Your Learning Universe

Siksha Sarovar is a free e-learning platform for coding courses, BCA university notes and competitive exam preparation. Optional Google sign-in saves your learning progress across devices.

Initializing knowledge base…
Compiling modules 0%

Unit 2.5: Comparison & Summary

Lesson 11 of 22 in the free Engineering Optimization notes on Siksha Sarovar, written by Rohit Jangra.

Unit 2.5: Comparison of Algorithms

This comparative analysis is essential for selecting the right method for a given engineering problem.

1. Comparison Table

FeatureSteepest DescentNewton’s MethodConjugate Gradient
OrderFirst-Order (Gradient only)Second-Order (Gradient + Hessian)First-Order (Gradient history)
ConvergenceLinear (Slow)Quadratic (Very Fast)Super-linear (Fast)
DirectionNegative Gradient (-∇f)Modified by Hessian (-[J]⁻¹∇f)Conjugate Direction (S + βS_old)
ComputationLow per iterationHigh (Matrix Inversion)Moderate
MemoryLow (Vector only)High (Store n*n Matrix)Low (Current & Prev Vectors)
StabilityAlways improvesCan diverge if Hessian badStable with good line search
Best ForSimple problems; Initial stepsComplex, non-linear; Final stepsLarge-scale problems (e.g., FEM)

2. Key Definition Summary

  • Gradient Vector (∇f): Points to steepest ascent. We go opposite to it.
  • Hessian Matrix ([J] or H): Describes the curvature (bowl shape).
  • Positive Definite: If the Hessian is positive definite, the function is strictly convex (like a nice bowl), and Newton's method works perfectly.
  • Step Size (λ): A scalar multiplier that decides how far to jump in the search direction. If too small, convergence is slow. If too large, we might overshoot the minimum.