Text Preprocessing — Lemmatization

Lemmatization reduces a word to its lemma — the dictionary/base form — using vocabulary and morphological (grammatical) analysis, unlike stemming's crude suffix-chopping.

"studies"  --lemmatization-->  "study"
"better"   --lemmatization-->  "good"     (stemming could never do this!)
"was"      --lemmatization-->  "be"

Lemmatization with NLTK (WordNet)

import nltk
from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()

print(lemmatizer.lemmatize("studies"))          # study      (default pos='n', noun)
print(lemmatizer.lemmatize("studies", pos='v'))  # study      (as a verb)
print(lemmatizer.lemmatize("better", pos='a'))   # good       (as an adjective)
print(lemmatizer.lemmatize("running", pos='v'))  # run
print(lemmatizer.lemmatize("running"))           # running    (WRONG -- default pos is noun!)

Critical detail: WordNet's lemmatizer needs to know the part of speech (POS) to lemmatize correctly. Without it, lemmatize("running") treats "running" as a noun and returns it unchanged.

Automatically Supplying POS Tags

from nltk import pos_tag, word_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet

lemmatizer = WordNetLemmatizer()

def get_wordnet_pos(tag):
    if tag.startswith('J'):
        return wordnet.ADJ
    elif tag.startswith('V'):
        return wordnet.VERB
    elif tag.startswith('N'):
        return wordnet.NOUN
    elif tag.startswith('R'):
        return wordnet.ADV
    return wordnet.NOUN   # default

sentence = "The children were running and playing happily"
tokens = word_tokenize(sentence)
tagged = pos_tag(tokens)

lemmas = [lemmatizer.lemmatize(word, get_wordnet_pos(tag)) for word, tag in tagged]
print(lemmas)
# ['The', 'child', 'be', 'run', 'and', 'play', 'happily']

Lemmatization with spaCy (Production-Grade, Context-Aware)

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("The children were running and playing happily")
print([token.lemma_ for token in doc])
# ['the', 'child', 'be', 'run', 'and', 'play', 'happily']

spaCy handles POS-tagging internally, so it does not require manually mapping POS tags.

Stemming vs Lemmatization — Direct Comparison

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

words = ["studies", "better", "caring", "was"]
pos_for_words = ['n', 'a', 'v', 'v']

for w, p in zip(words, pos_for_words):
    print(f"{w:10} | stem: {stemmer.stem(w):8} | lemma: {lemmatizer.lemmatize(w, p)}")

# studies    | stem: studi    | lemma: study
# better     | stem: better   | lemma: good
# caring     | stem: care     | lemma: care
# was        | stem: wa       | lemma: be

When to Use Which