Building a Complete Text Preprocessing Pipeline

Now that we've covered tokenization, normalization, stemming, lemmatization, stop-word removal, and regex, let's assemble them into a single, reusable end-to-end pipeline — exactly what you would build before feeding text into a classifier, search index, or model in later units.

The Full Pipeline in Python

import re
import string
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from nltk import pos_tag
from nltk.corpus import wordnet

lemmatizer = WordNetLemmatizer()
stop_words = set(stopwords.words('english'))

def get_wordnet_pos(tag):
    if tag.startswith('J'):
        return wordnet.ADJ
    elif tag.startswith('V'):
        return wordnet.VERB
    elif tag.startswith('R'):
        return wordnet.ADV
    return wordnet.NOUN

def preprocess(text):
    # 1. Lowercase
    text = text.lower()
    # 2. Remove URLs and mentions
    text = re.sub(r'http\S+|@\w+', '', text)
    # 3. Remove punctuation and digits
    text = text.translate(str.maketrans('', '', string.punctuation))
    text = re.sub(r'\d+', '', text)
    # 4. Tokenize
    tokens = word_tokenize(text)
    # 5. Remove stop words
    tokens = [t for t in tokens if t not in stop_words]
    # 6. Lemmatize using POS tags
    tagged = pos_tag(tokens)
    lemmas = [lemmatizer.lemmatize(w, get_wordnet_pos(p)) for w, p in tagged]
    return lemmas

raw = "Check out https://example.com! @NLPFan said the courses are AMAZING and REALLY helpful in 2026."
print(preprocess(raw))
# ['check', 'course', 'amazing', 'really', 'helpful']

Same Pipeline with spaCy (Fewer Steps, Production-Style)

import spacy
import re

nlp = spacy.load("en_core_web_sm")

def preprocess_spacy(text):
    text = re.sub(r'http\S+|@\w+', '', text)
    doc = nlp(text.lower())
    return [token.lemma_ for token in doc
            if not token.is_stop and not token.is_punct and not token.is_space]

print(preprocess_spacy("Check out https://example.com! @NLPFan said the courses are AMAZING."))
# ['check', 'course', 'amazing']

Preprocessing Checklist — Decide Per Task

There is no single "correct" pipeline — the right combination of steps depends entirely on what you build next: search/BoW (Unit 3), a parser/tagger (Unit 2), or a modern embedding-based model (Unit 4).