Text Preprocessing — Normalization

Text normalization converts text into a single, consistent, canonical form so that different surface variations of the "same" content are treated identically by downstream models.

Common Normalization Steps

Lowercasing

text = "Natural Language Processing is FUN!"
print(text.lower())
# natural language processing is fun!

Removing Punctuation

import string

text = "Hello!!! Are you there??"
cleaned = text.translate(str.maketrans('', '', string.punctuation))
print(cleaned)
# Hello Are you there

Removing Numbers and Extra Whitespace

import re

text = "Order  #12345   was placed on   02/08/2026."
no_numbers = re.sub(r'\d+', '', text)
single_spaced = re.sub(r'\s+', ' ', no_numbers).strip()
print(single_spaced)
# Order # was placed on //.

Expanding Contractions

contractions = {
    "don't": "do not", "isn't": "is not", "it's": "it is",
    "can't": "cannot", "i'm": "i am", "you're": "you are"
}

def expand_contractions(text):
    words = text.lower().split()
    return " ".join(contractions.get(w, w) for w in words)

print(expand_contractions("I don't think it's ready"))
# i do not think it is ready

Unicode Normalization

import unicodedata

text = "café naïve résumé"
normalized = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8')
print(normalized)
# cafe naive resume

A Combined Normalization Function

import re
import string

def normalize(text):
    text = text.lower()
    text = re.sub(r'\d+', '', text)                              # remove numbers
    text = text.translate(str.maketrans('', '', string.punctuation))  # remove punctuation
    text = re.sub(r'\s+', ' ', text).strip()                     # collapse whitespace
    return text

raw = "  The PRICE is Rs. 1500!!  Call NOW @9999999999  "
print(normalize(raw))
# the price is rs call now

Caveat — Normalization Is Task-Dependent

Aggressive normalization is not always correct: removing numbers would destroy a model meant to extract phone numbers, and lowercasing can hurt Named Entity Recognition (where capitalization is a strong signal that "Apple" is a company, not a fruit). Always tailor normalization to the downstream task.