13. Learning Neural Network Structures

The architecture and regularization of a neural network are just as important as the training algorithm itself. This lesson covers practical techniques for designing networks that generalize well.

Regularization Techniques

L1 and L2 Weight Decay

Adding a penalty to the loss function prevents large weights:

• L2 (Ridge/Weight Decay): L_reg = L + (lambda/2) * ||W||^2 — shrinks all weights toward zero, smooth solutions
• L1 (Lasso): L_reg = L + lambda * ||W||_1 — drives many weights to exactly zero, promotes sparsity

Dropout (Srivastava et al., 2014)

During training, randomly set each neuron's output to zero with probability p (typically 0.5 for hidden layers, 0.2 for input):

• Prevents co-adaptation of neurons
• Equivalent to training an ensemble of 2^n thinned networks
• At test time, scale activations by (1-p) (or use inverted dropout)

Batch Normalization (Ioffe and Szegedy, 2015)

Normalize activations within each mini-batch: BN(x) = gamma * (x - mu_batch) / sqrt(sigma_batch^2 + epsilon) + beta

Benefits: reduces internal covariate shift, allows higher learning rates, mild regularization effect.

Architecture Design Guidelines

Hyperparameter Search Strategies

1. Start with reasonable defaults (e.g., lr=0.001, Adam optimizer, dropout=0.3)
2. Use learning rate range test to find good learning rate
3. Random search > grid search for > 3 hyperparameters
4. Use validation loss as criterion, not training loss

Common Pitfalls

• Dropout during inference will give wrong results — always disable at test time
• Batch normalization interacts with dropout — use BN before dropout
• Very wide networks can overfit even with regularization; depth + regularization is often better

Exam-Ready Summary

• L2 regularization: shrinks weights; L1: promotes sparsity
• Dropout: randomly zero activations during training, ensemble interpretation
• Batch normalization: normalize within mini-batch; reduces covariate shift
• Architecture: more depth generally beats more width for complex tasks
• Hyperparameter tuning: random search is practical; Bayesian optimization for expensive searches