9. Factor Analysis

Factor Analysis (FA) is a probabilistic generative model that explains observed variables as linear combinations of a small number of latent factors plus unique noise. Developed in psychometrics (Spearman, 1904), FA differs from PCA by explicitly modeling measurement noise.

Generative Model

x = W * z + mu + epsilon

Where:

• x: observed p-dimensional variable
• z: latent factor (k-dimensional, Gaussian: z ~ N(0, I))
• W: factor loading matrix (p x k)
• epsilon: unique noise (epsilon ~ N(0, Psi), Psi diagonal)
• mu: mean of observed variable

FA vs PCA

Parameter Estimation

Parameters W and Psi are estimated by maximum likelihood using the EM algorithm:

• E-step: Compute posterior of latent factors given observed data
• M-step: Update W and Psi to maximize expected log-likelihood

The observed data covariance is modeled as: Sigma = W * W^T + Psi

Rotation Methods

Since FA is only identifiable up to orthogonal rotation, rotation methods improve interpretability:

• Varimax: Maximizes variance of squared loadings (each factor loads heavily on few variables)
• Oblimin: Allows correlated factors

Common Pitfalls

• FA requires specifying k (number of factors) in advance — use scree plot or BIC
• The EM algorithm may converge to local optima — use multiple random initializations
• Heywood cases: negative unique variances indicate model misspecification

Exam-Ready Summary

• Factor Analysis: generative model where observed = linear combination of latent factors + noise
• Key difference from PCA: FA explicitly models unique noise per variable
• FA assumes data covariance = W*W^T + Psi (low-rank + diagonal)
• Parameters estimated by EM (maximum likelihood)
• Rotation (varimax, oblimin) improves factor interpretability after fitting