Part-of-Speech (POS) Tagging

POS tagging is the process of assigning a grammatical category (noun, verb, adjective, etc.) to every token in a sentence, based on both its definition and its context.

"Riya reads books."
# Riya  -> Proper Noun
# reads -> Verb
# books -> Noun
# .     -> Punctuation

Why POS Tagging Matters

POS tags are a foundational feature used by almost every later stage of NLP:

• Parsing needs POS tags to build a syntax tree (Unit 2, next lessons)
• Lemmatization needs POS to pick the right lemma (recall Unit 1: "running" → "run" only if tagged as a verb)
• NER uses POS patterns (e.g. sequences of proper nouns) as features
• Word Sense Disambiguation narrows down possible senses using POS

The Penn Treebank Tagset (Most Common in English NLP)

Tagging with NLTK

import nltk
from nltk import pos_tag, word_tokenize

sentence = "Riya quickly finished the difficult assignment."
tokens = word_tokenize(sentence)
tagged = pos_tag(tokens)
print(tagged)
# [('Riya', 'NNP'), ('quickly', 'RB'), ('finished', 'VBD'), ('the', 'DT'),
#  ('difficult', 'JJ'), ('assignment', 'NN'), ('.', '.')]

Why POS Tagging Is Hard — Ambiguity

The same word form can take different tags depending on context:

sentences = [
    "I book a flight ticket.",      # "book" -> VB (verb)
    "I read a good book.",          # "book" -> NN (noun)
]
for s in sentences:
    print(pos_tag(word_tokenize(s)))
# [('I', 'PRP'), ('book', 'VBP'), ('a', 'DT'), ('flight', 'NN'), ('ticket', 'NN'), ('.', '.')]
# [('I', 'PRP'), ('read', 'VBP'), ('a', 'DT'), ('good', 'JJ'), ('book', 'NN'), ('.', '.')]

This is the same fundamental problem as lexical ambiguity from Unit 1 — a single surface word form maps to multiple grammatical categories, and only context disambiguates it. Resolving this correctly is exactly what POS tagging algorithms — covered in the next lesson — are built to do.