6. Linear Discriminant Analysis (LDA)

Linear Discriminant Analysis was introduced by R.A. Fisher in 1936 as a method to find the linear combination of features that best separates two or more classes. Unlike PCA (which ignores class labels), LDA is a supervised dimensionality reduction technique that maximizes class separability.

Fisher's Criterion

LDA finds a projection W that maximizes the ratio of between-class scatter to within-class scatter: J(W) = |W^T S_B W| / |W^T S_W W|

Where:

• S_B = between-class scatter matrix = sum_c n_c * (mu_c - mu)(mu_c - mu)^T
• S_W = within-class scatter matrix = sum_c sum_{x in c} (x - mu_c)(x - mu_c)^T

The optimal W consists of eigenvectors of S_W^{-1} * S_B.

LDA vs PCA

LDA Assumptions

1. Gaussian class distributions: Each class is multivariate Gaussian.
2. Equal class covariances: All classes share the same covariance matrix (homoscedasticity).
3. Linear separability: Classes are linearly separable in the projected space.

When assumptions hold, LDA = Bayes optimal classifier for Gaussian classes.

Worked Example: Wine Dataset

The UCI Wine dataset has 13 features and 3 classes. LDA projection onto 2 components achieves ~98% KNN accuracy vs ~82% for PCA with 2 components, because LDA's 2 components are chosen for class separation, not maximum variance.

Common Pitfalls

• LDA fails when within-class covariances differ significantly (use QDA instead)
• Degenerate S_W when n_features > n_samples (use regularized LDA)
• Sensitive to outliers which distort scatter matrices

Exam-Ready Summary

• LDA: supervised, maximizes between-class/within-class scatter ratio
• Max components = C-1 for C-class problem (always fewer components than PCA)
• Assumes Gaussian classes with equal covariances (unlike QDA which allows different covariances)
• LDA is the Bayes optimal classifier under its assumptions
• Regularized LDA (RLDA) handles p > n by adding lambda*I to S_W