Introduction to Transformers

The Transformer architecture (Vaswani et al., "Attention Is All You Need", 2017) replaced RNNs/LSTMs (previous lesson) as the dominant architecture in NLP. Its key innovation is self-attention, which lets the model directly relate every word in a sequence to every other word — in parallel, without the sequential bottleneck of RNNs.

Why Self-Attention?

Recall the RNN limitation from the previous lesson: information from early words fades by the time the network reaches later words. Self-attention lets every word look directly at every other word in the sentence — regardless of distance — and learn which words matter most for understanding it.

"The animal didn't cross the street because it was too tired"
                                              ^
Self-attention lets "it" directly attend to "animal" (not "street"),
correctly resolving what "it" refers to -- this is coreference resolution
(recall Unit 1's discourse analysis!) learned automatically by the model.

The Core Self-Attention Mechanism (Conceptual)

For each word, the model computes three vectors: Query (Q), Key (K), and Value (V). Attention scores are computed by comparing each word's Query against every other word's Key, then using those scores to create a weighted combination of all Values.

Attention(Q, K, V) = softmax( Q · K^T / sqrt(d_k) ) · V

• The higher the similarity between a word's Query and another word's Key, the more that other word's Value contributes to the output representation.
• Multi-head attention runs this process several times in parallel with different learned projections, letting the model capture different types of relationships simultaneously (e.g. one head might track syntactic subject-verb links, another might track coreference).

Positional Encoding — Restoring Word Order

Unlike RNNs, self-attention has no inherent sense of word order (it looks at all words simultaneously). Transformers add a positional encoding vector to each word's embedding to inject information about its position in the sequence.

Encoder vs Decoder Transformers

Using a Pre-trained Transformer (BERT) with Hugging Face

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

sentence1 = "I deposited money at the bank."
sentence2 = "We sat on the river bank."

for sentence in [sentence1, sentence2]:
    inputs = tokenizer(sentence, return_tensors="pt")
    outputs = model(**inputs)
    bank_index = tokenizer.tokenize(sentence).index("bank")
    bank_vector = outputs.last_hidden_state[0][bank_index + 1]  # +1 for [CLS] token
    print(sentence, "-> bank vector (first 5 dims):", bank_vector[:5])
# The two "bank" vectors will be DIFFERENT -- unlike Word2Vec's single fixed vector,
# BERT produces CONTEXTUAL embeddings that resolve word sense (WSD, Unit 2) automatically!

Word2Vec vs Contextual (Transformer) Embeddings

Transformer Applications Across This Course

Nearly every task covered in this course now has a Transformer-based state-of-the-art solution: POS tagging and NER (Unit 2) via BERT-based token classifiers, sentiment analysis and text classification (Unit 3) via fine-tuned BERT/RoBERTa, and abstractive summarization (Unit 3) via BART/T5. The Transformer architecture is also the direct foundation of the Large Language Models covered in the next lesson.