1. Evaluating ML Algorithms and Model Selection

Model evaluation is the foundation of trustworthy ML. Without rigorous evaluation, we cannot distinguish genuine generalization from overfitting — a model that memorizes training data but fails in production.

Bias-Variance Decomposition

The generalization error decomposes as: Total Error = Bias^2 + Variance + Irreducible Noise

• Bias: Error from incorrect assumptions. High bias implies underfitting — the model is too simple.
• Variance: Error from sensitivity to training data. High variance implies overfitting — the model is too complex.
• Sweet spot: Complexity that minimizes bias^2 + variance simultaneously.

Cross-Validation Techniques

Classification Metrics

Model Selection Strategies

1. Grid Search: Exhaustive sweep over hyperparameter grid.
2. Random Search: Sample from distributions — often 3x more efficient for large grids.
3. Bayesian Optimization: Build a surrogate model to focus on promising hyperparameter regions.

Common Pitfalls

• Data leakage: Preprocessing using future information contaminates results.
• Test set reuse: Multiple comparisons inflate apparent performance.
• Wrong metric: Accuracy is misleading on highly imbalanced datasets.

Exam-Ready Summary

• 5-fold or 10-fold CV is standard; LOOCV is unbiased but expensive
• Always stratify folds for classification tasks
• High bias: increase model complexity or add features
• High variance: add data, apply regularization, reduce model complexity
• Data leakage is the most dangerous source of misleading evaluation results