Text Preprocessing — Tokenization
Raw text is unstructured and noisy. Before any NLP model can use it, text must go through a preprocessing pipeline. The first step is almost always tokenization.
What is Tokenization?
Tokenization is the process of splitting text into smaller units called tokens — usually words, subwords, or sentences.
| Type | Splits Into | Example |
|---|---|---|
| Word tokenization | Individual words/punctuation | "NLP is fun!" → ["NLP", "is", "fun", "!"] |
| Sentence tokenization | Individual sentences | "Hi. How are you?" → ["Hi.", "How are you?"] |
| Subword tokenization | Word pieces (used by modern LLMs) | "unhappiness" → ["un", "happiness"] |
Word Tokenization
import nltk
from nltk.tokenize import word_tokenize
text = "Mr. Sharma isn't going to Delhi; he's flying to Mumbai."
tokens = word_tokenize(text)
print(tokens)
# ['Mr.', 'Sharma', 'is', "n't", 'going', 'to', 'Delhi',
# ';', 'he', "'s", 'flying', 'to', 'Mumbai', '.']Notice how word_tokenize correctly:
- Keeps "Mr." together (does not split on the period after an abbreviation)
- Splits "isn't" into "is" + "n't" (handles contractions)
- Treats punctuation (";", ".") as separate tokens
Sentence Tokenization
from nltk.tokenize import sent_tokenize
paragraph = "NLP is exciting. It powers chatbots, search engines, and more! Are you ready to learn?"
sentences = sent_tokenize(paragraph)
print(sentences)
# ['NLP is exciting.', 'It powers chatbots, search engines, and more!', 'Are you ready to learn?']Why Simple .split() Is Not Enough
text = "Dr. Riya's report costs Rs. 1,50,000."
print(text.split())
# ['Dr.', "Riya's", 'report', 'costs', 'Rs.', '1,50,000.']
# Wrong: "Dr." kept with the period attached, punctuation not separated,
# and numbers with commas are mangled -- naive .split() has no linguistic rules.Whitespace, Punctuation & Regex-based Tokenizers
from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r'\w+') # keeps only alphanumeric sequences
print(tokenizer.tokenize("NLP's great, isn't it?"))
# ['NLP', 's', 'great', 'isn', 't', 'it']Challenges in Tokenization
- Abbreviations: "U.S.A." — is each period a sentence boundary?
- Contractions: "don't", "I'll" — split into how many tokens?
- Hyphenated words: "state-of-the-art" — one token or four?
- Numbers with punctuation: "3.14", "1,000", dates like "01/02/2026"
- No word boundaries: languages like Chinese/Japanese have no spaces between words at all — tokenization requires a dictionary or statistical model.
- Social media text: hashtags, emojis, "@mentions", "gr8", "lol" need special handling.
Tokenization sets the foundation for every downstream step — an incorrect tokenizer choice propagates errors through the entire pipeline.