Unit I Overview: Combining Different Models

Ensemble methods combine multiple base learners to produce a more powerful predictor. Condorcet's Jury Theorem (1785) shows that when independent voters are slightly better than chance, a majority vote approaches certainty. Machine learning exploits this through bagging, boosting, and stacking.

Why Ensembles Work

The expected test error decomposes as: Total Error = Bias^2 + Variance + Irreducible Noise

• Bagging trains models in parallel on bootstrap samples — reduces variance by averaging.
• Boosting trains models sequentially, correcting previous errors — reduces bias.
• Stacking learns a meta-model to optimally weight diverse base learners.

Unit I Roadmap

No Free Lunch Theorem

No single algorithm outperforms all others on every problem distribution. This motivates ensembles: diverse models cover different hypothesis regions for broadly robust performance.

Requirements for Effective Ensembles

1. Base learner accuracy must exceed 50% (better than random).
2. Base learners must make diverse, uncorrelated errors — diversity is paramount.
3. A combination rule (vote, average, or meta-learner) must aggregate predictions.

Exam-Ready Summary

• Ensemble error = f(individual errors, inter-model correlation)
• Lower inter-model correlation means greater variance reduction
• Bagging: parallel, best with unstable high-variance models (deep trees)
• Boosting: sequential, best with stable weak learners (decision stumps)
• Diversity is essential — identical models provide zero ensemble benefit