10. Independent Component Analysis (ICA)

ICA is a computational technique for separating a multivariate signal into statistically independent non-Gaussian components. The classic motivating example is the cocktail party problem: given recordings from microphones placed in a room with multiple speakers, recover the individual speakers' voices.

Problem Formulation

Observed signal: x = A * s

• x: observed mixtures (p-dimensional)
• A: unknown mixing matrix (p x p)
• s: unknown independent source signals (p-dimensional)

Goal: find W such that s_hat = W * x recovers independent sources.

Key Assumptions

1. Source signals s_i are statistically independent.
2. At most one source can be Gaussian (a fundamental constraint of ICA).
3. The mixing is instantaneous (no time delays).

ICA vs PCA

FastICA Algorithm

1. Center and whiten data (make uncorrelated with unit variance)
2. For each component i:

a. Initialize random weight vector w b. Iterate: w = E[x G(w^Tx)] - E[G(w^Tx)] w c. Deflate: orthogonalize w against previous components

3. Sources: s_hat = W * x

Where G is the derivative of a non-linearity (e.g., tanh for kurtosis maximization).

Common Pitfalls

• ICA cannot determine order or scale of recovered sources (inherent ambiguity)
• Requires all sources to be non-Gaussian — fails if sources are Gaussian
• Computationally intensive for large datasets

Exam-Ready Summary

• ICA: recovers statistically independent sources from observed linear mixtures
• PCA decorrelates (2nd order); ICA achieves full statistical independence (all orders)
• FastICA uses kurtosis or negentropy as a measure of non-Gaussianity
• Order and scale of components are ambiguous — not a limitation in practice
• Applications: EEG/fMRI blind source separation, audio signal processing