Word Embeddings — Word2Vec
Recall from Unit 3 that Bag-of-Words / TF-IDF treats every word as an independent dimension — "good" and "great" share no similarity in that representation, even though they mean almost the same thing. Word embeddings solve this by representing each word as a dense, low-dimensional vector (typically 100–300 dimensions) where semantically similar words end up close together in vector space.
The Distributional Hypothesis
Word2Vec is built on a core linguistic idea: "a word is characterized by the company it keeps" (J.R. Firth) — words that appear in similar contexts tend to have similar meanings.
"The cat sat on the mat"
"The dog sat on the mat"
-> "cat" and "dog" appear in the same context -> Word2Vec learns similar vectors for themTwo Word2Vec Architectures
| Architecture | Predicts | Best for |
|---|---|---|
| CBOW (Continuous Bag of Words) | The target word, given its surrounding context words | Faster training, works well with frequent words |
| Skip-gram | The surrounding context words, given the target word | Works better with rare words, small datasets |
Training Word2Vec with Gensim
from gensim.models import Word2Vec
sentences = [
["the", "cat", "sat", "on", "the", "mat"],
["the", "dog", "sat", "on", "the", "mat"],
["cats", "and", "dogs", "are", "great", "pets"],
["the", "king", "loves", "the", "queen"],
["the", "queen", "loves", "the", "king"],
]
model = Word2Vec(sentences, vector_size=50, window=2, min_count=1, sg=1) # sg=1 -> skip-gram
vector = model.wv["cat"]
print(vector.shape) # (50,) -- a dense 50-dimensional vector
similar = model.wv.most_similar("cat", topn=3)
print(similar)
# [('dog', 0.98...), ('mat', 0.75...), ('sat', 0.60...)] (scores illustrative on tiny toy corpus)The Famous Vector Arithmetic Property
Word2Vec embeddings capture analogies through simple vector arithmetic — this only works well on models trained on very large corpora (e.g. Google News, 100 billion+ words):
# king - man + woman ≈ queen (illustrative; requires a large pre-trained model)
result = model.wv.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
print(result)
# [('queen', 0.73)]Using Pre-trained Word2Vec Embeddings
Training from scratch needs huge corpora; in practice, most projects load pre-trained embeddings:
import gensim.downloader as api
wv = api.load("word2vec-google-news-300") # 300-dim vectors trained on ~100B words
print(wv.similarity("good", "great")) # 0.72 (high -- semantically close)
print(wv.similarity("good", "bad")) # 0.19 (low -- semantically distant, despite being frequent co-occurring antonyms)
print(wv.most_similar("python"))
# Mix of programming-related and snake-related words -- WSD-style ambiguity (Unit 2) still applies!Word2Vec vs BoW/TF-IDF
| BoW / TF-IDF (Unit 3) | Word2Vec | |
|---|---|---|
| Dimensionality | High (= vocabulary size) | Low (typically 100-300) |
| Sparsity | Sparse (mostly zeros) | Dense |
| Captures word order | No | Partially (via context window) |
| Captures semantic similarity | No | Yes |
| Out-of-vocabulary words | Simply absent (0 count) | Cannot be represented (fixed vocabulary) — solved by later methods like FastText and subword tokenization |
Other Embedding Techniques (Brief Mention)
| Method | Key Idea |
|---|---|
| GloVe | Learns embeddings from global word co-occurrence statistics across the whole corpus |
| FastText | Extends Word2Vec using character n-grams — can represent out-of-vocabulary/misspelled words |
| Contextual embeddings (BERT, etc.) | A word gets a different vector depending on its sentence context (unlike Word2Vec's one fixed vector per word) — covered next, as the foundation of Transformers |
Word2Vec was a landmark 2013 breakthrough — but it still assigns one fixed vector per word, regardless of context (so "bank" the river and "bank" the financial institution get the same vector). Solving this limitation is exactly what motivates the Transformer architecture, covered in the next two lessons.