Introduction to Deep Learning for NLP

Classical NLP methods (n-gram LMs in Unit 2, Naïve Bayes with TF-IDF in Unit 3, Word2Vec in the previous lesson) hit fundamental limits: fixed context windows, no ability to model long-range dependencies, and — for Word2Vec — one fixed vector per word regardless of context. Deep learning models address these by learning hierarchical, context-aware representations directly from data.

Why Neural Networks for NLP?

Recurrent Neural Networks (RNNs)

An RNN processes a sequence of tokens one at a time, maintaining a hidden state that is updated at every step and carries information forward — a natural fit for language, which is inherently sequential.

h_t = f(W_x · x_t + W_h · h_{t-1} + b)

Each hidden state h_t is a function of the current input x_t AND the previous hidden state h_{t-1} — this is how the network "remembers" earlier words while reading a sentence.

The Vanishing Gradient Problem

Plain RNNs struggle to learn long-range dependencies — when a sentence is long, gradients used during training shrink exponentially as they are propagated backward through many time steps, so the network effectively "forgets" information from many words ago.

"The keys, which were on the table near the door that Riya had painted last summer, ___"
                                                                                  ^
A plain RNN struggles to remember "keys" (singular subject!) is far back --> may
incorrectly predict "were" instead of "was", losing the singular/plural agreement.

LSTM (Long Short-Term Memory) — Solving Vanishing Gradients

LSTM networks introduce a separate cell state and three gates (forget, input, output) that explicitly control what information to keep, add, or discard at each time step — allowing the network to preserve important information over much longer sequences.

GRU (Gated Recurrent Unit)

A simplified variant of LSTM with only two gates (reset and update) — fewer parameters, faster to train, and often comparable performance to LSTM.

Building a Simple Text Classifier with an LSTM (Keras)

import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense

texts = ["great product loved it", "terrible quality very bad",
         "excellent value amazing", "worst purchase ever"]
labels = [1, 0, 1, 0]   # 1 = positive, 0 = negative

tokenizer = Tokenizer(num_words=1000)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
padded = pad_sequences(sequences, maxlen=6)

model = Sequential([
    Embedding(input_dim=1000, output_dim=16, input_length=6),
    LSTM(32),
    Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(padded, labels, epochs=10, verbose=0)

Bidirectional RNNs — Reading Context from Both Directions

A Bidirectional LSTM (BiLSTM) runs two LSTMs over the sequence — one forward, one backward — and combines both, so every word's representation is informed by words both before and after it. This is exactly why BiLSTM-based taggers/NER models (mentioned back in Unit 2) outperform plain left-to-right models: knowing what comes after a word often disambiguates its role.

From RNNs to Transformers

RNNs (and LSTMs/GRUs) still process text sequentially, one token at a time — which is slow to train (cannot fully parallelize) and still struggles with very long-range dependencies despite gating. The next lesson introduces the Transformer architecture, which replaced RNNs as the dominant approach by processing an entire sequence in parallel using a mechanism called self-attention.