Text Preprocessing — Stemming
Stemming reduces a word to its root/stem form by chopping off suffixes (and sometimes prefixes) using heuristic rules — without necessarily producing a valid dictionary word.
"studies", "studying", "studied" --stemming--> "studi"
"connection", "connected", "connecting" --stemming--> "connect"Why Stem?
Stemming reduces vocabulary size by collapsing inflected/derived forms of a word into one representative token, which helps tasks like search and information retrieval treat "run", "running", and "runs" as the same underlying concept.
Porter Stemmer (Most Widely Used)
The Porter Stemmer (Martin Porter, 1980) applies a sequence of rule-based suffix-stripping steps.
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
words = ["studies", "studying", "studied", "connection", "connected", "connecting", "happiness"]
for w in words:
print(w, "->", stemmer.stem(w))
# studies -> studi
# studying -> studi
# studied -> studi
# connection -> connect
# connected -> connect
# connecting -> connect
# happiness -> happiNotice stemmer.stem("happiness") gives "happi" — not a real English word. This is expected; stemming trades linguistic correctness for speed and simplicity.
Snowball Stemmer (Improved Porter2, Multi-language)
from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
print(stemmer.stem("generously")) # generous
print(stemmer.stem("running")) # run
# Snowball also supports many other languages:
print(SnowballStemmer.languages)
# ('arabic', 'danish', 'dutch', 'english', 'finnish', 'french',
# 'german', 'hungarian', ... )Lancaster Stemmer (Very Aggressive)
from nltk.stem import LancasterStemmer
lancaster = LancasterStemmer()
print(lancaster.stem("maximum")) # maxim
print(lancaster.stem("presumably")) # presumComparing Stemmers
from nltk.stem import PorterStemmer, LancasterStemmer
porter = PorterStemmer()
lancaster = LancasterStemmer()
word = "generously"
print("Porter: ", porter.stem(word)) # gener
print("Lancaster:", lancaster.stem(word)) # gen
# Lancaster is faster but MORE aggressive -> more information lossProblems with Stemming
| Problem | Example |
|---|---|
| Over-stemming | Two unrelated words reduced to the same stem: "university" and "universe" both → "univers" |
| Under-stemming | Two related words NOT reduced to the same stem: "alumnus" and "alumni" stay different |
| Produces non-words | "happiness" → "happi" is not a valid English word |
Stemming vs Lemmatization (Preview)
| Stemming | Lemmatization | |
|---|---|---|
| Approach | Rule-based suffix stripping | Dictionary + grammar-aware |
| Speed | Fast | Slower |
| Output | May not be a real word ("happi") | Always a valid dictionary word ("happy") |
| Accuracy | Lower | Higher |
We cover lemmatization — the more linguistically accurate alternative — in the next lesson.