Introduction to Language Models

A language model (LM) is a system that assigns a probability to a sequence of words — modeling how likely that sequence is to occur in the language.

P("I love natural language processing")   -> high probability (fluent, meaningful)
P("processing love I natural language")   -> low probability  (word salad)

What Language Models Are Used For

Formal Definition

For a sentence W = w1, w2, ..., wn, a language model computes:

P(W) = P(w1, w2, ..., wn)

Using the chain rule of probability:

P(w1, w2, ..., wn) = P(w1) · P(w2|w1) · P(w3|w1,w2) · ... · P(wn|w1,...,w_{n-1})

Computing P(wn | w1, ..., w_{n-1}) exactly requires seeing that exact history in training data — which becomes impossible for long sentences (data sparsity). This motivates the n-gram approximation, covered in detail in the next lesson.

The Markov Assumption

Instead of conditioning on the entire history, we assume a word depends only on the previous N-1 words:

Unigram (N=1): P(wi)                          -- ignores all context
Bigram  (N=2): P(wi | w_{i-1})                -- depends on 1 previous word
Trigram (N=3): P(wi | w_{i-2}, w_{i-1})       -- depends on 2 previous words

This is called the Markov assumption, and it is what makes language modeling computationally tractable.

A Tiny Bigram Model by Hand

from collections import defaultdict, Counter
from nltk.tokenize import word_tokenize

corpus = "I love NLP . I love Python . NLP is fun ."
tokens = word_tokenize(corpus.lower())

bigrams = list(zip(tokens, tokens[1:]))
bigram_counts = Counter(bigrams)
unigram_counts = Counter(tokens)

def bigram_prob(w1, w2):
    return bigram_counts[(w1, w2)] / unigram_counts[w1]

print(bigram_prob("i", "love"))   # 2/2 = 1.0  (both times "i" is followed by "love")
print(bigram_prob("love", "nlp")) # 1/2 = 0.5  ("love" is followed by "nlp" once, "python" once)

Statistical vs Neural Language Models

N-gram models are the classical, foundational approach — we build one fully, including handling unseen words via smoothing, over the next two lessons. Neural language models (word embeddings, Transformers, LLMs) are the subject of Unit 4.