14. Deep Learning and Feature Representation Learning

Deep learning is characterized by learning hierarchical feature representations directly from raw data. Rather than hand-crafting features, deep networks automatically discover the representations needed for detection or classification, building increasingly abstract concepts layer by layer.

Why Depth Matters

In a deep network, each layer transforms the representation learned by the previous layer:

• Layer 1 (images): edges, corners, blobs
• Layer 2: textures, patterns, simple shapes
• Layer 3: object parts (eyes, wheels, windows)
• Layer 4: whole objects (faces, cars)

Mathematically, depth provides compositional representations: deep circuits can represent exponentially more functions efficiently than shallow ones.

Convolutional Neural Networks (CNNs)

CNNs exploit the spatial structure of images through:

• Convolution layers: Local feature detectors (filters/kernels) applied with weight sharing
• Pooling layers: Spatial downsampling (max pooling most common) — translation invariance
• Fully connected layers: Final classification head

Key insight: parameter count = (kernel_h kernel_w in_channels + 1) * out_channels

Dramatically fewer parameters than fully connected layers on image data.

Recurrent Neural Networks (RNNs)

For sequential data, RNNs maintain a hidden state: h_t = tanh(W_h h_{t-1} + W_x x_t + b)

LSTMs solve the vanishing gradient over time via gating mechanisms (forget, input, output gates).

Transfer Learning

Pre-train on large dataset (e.g., ImageNet with 1M+ images), then fine-tune on target task:

1. Load pre-trained weights (e.g., ResNet, VGG, BERT)
2. Freeze lower layers (general features)
3. Fine-tune upper layers + new classification head

ImageNet Milestones

Exam-Ready Summary

• Deep learning: hierarchical feature learning — lower layers detect primitives, higher layers detect concepts
• CNNs: weight sharing + local connectivity = efficient spatial feature extraction
• ResNets (skip connections) solve the degradation problem in very deep networks
• Transfer learning: pre-train on large data, fine-tune on small target dataset
• Batch normalization + ReLU + skip connections are key ingredients of modern architectures