16. Generalization in Reinforcement Learning

Generalization in RL is the ability to perform well on states not seen during training. Deep RL extends the Q-function and policy to neural networks, enabling agents to operate in large, continuous state spaces.

Deep Q-Networks (DQN)

DeepMind (Mnih et al., 2015) achieved human-level performance on 49 Atari games using: Q(s, a; theta) ≈ Q*(s, a) (parameterized by neural network)

Two key innovations:

1. Experience Replay: Store transitions (s, a, r, s') in a replay buffer. Sample random mini-batches to break temporal correlations between consecutive samples.

1. Target Network: Use a separate, periodically updated target network to compute stable TD targets:

L(theta) = E[(r + gamma * max_a' Q(s', a'; theta^-) - Q(s, a; theta))^2]

Deadly Triad in Function Approximation

Three ingredients that together can cause divergence:

1. Off-policy learning: Learning from data generated by a different policy
2. Function approximation: Approximating Q with a non-linear function (e.g., neural network)
3. Bootstrapping: Using current estimates to update current estimates (TD methods)

Experience replay and target networks stabilize the otherwise divergent deadly triad.

Generalization Challenges

Policy Gradient Methods

Instead of learning Q, directly optimize the policy: J(theta) = E_pi[G_t], gradient: nabla J = E[nabla log pi(a|s) * G_t]

This REINFORCE estimator is high variance — Actor-Critic methods reduce variance using a baseline.

Exam-Ready Summary

• DQN: Q-learning with neural network function approximator + experience replay + target network
• Experience replay: breaks temporal correlations, improves sample efficiency
• Target network: prevents oscillations by fixing Q-learning targets for several steps
• Deadly triad: off-policy + function approximation + bootstrapping can diverge
• Actor-Critic: policy gradient with value function baseline to reduce variance