Text Classification — End-to-End Pipeline

This lesson combines everything from Units 1–3 so far — preprocessing, feature extraction, and Naïve Bayes — into one complete, realistic text classification project: classifying news headlines by topic.

Step 1 — The Dataset

headlines = [
    "Team wins championship in thrilling final match",
    "New smartphone launched with advanced AI camera",
    "Stock market rallies after positive earnings report",
    "Player scores hat-trick in league match",
    "Tech company unveils next-generation processor chip",
    "Investors cheer as markets hit record high",
]
labels = ["sports", "tech", "business", "sports", "tech", "business"]

Step 2 — Preprocessing (Recap from Unit 1)

import re
import string
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
stop_words = set(stopwords.words('english'))

def preprocess(text):
    text = text.lower()
    text = text.translate(str.maketrans('', '', string.punctuation))
    tokens = word_tokenize(text)
    tokens = [lemmatizer.lemmatize(t) for t in tokens if t not in stop_words]
    return " ".join(tokens)

cleaned = [preprocess(h) for h in headlines]
print(cleaned[0])
# 'team win championship thrilling final match'

Step 3 — Feature Extraction with TF-IDF

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(cleaned)
print(X.shape)   # (6 documents, N unique terms)

Step 4 — Train/Test Split and Model Training

from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB

X_train, X_test, y_train, y_test = train_test_split(
    X, labels, test_size=0.33, random_state=42
)

model = MultinomialNB()
model.fit(X_train, y_train)

Step 5 — Evaluation

from sklearn.metrics import accuracy_score, classification_report

predictions = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, predictions))
print(classification_report(y_test, predictions))
#               precision    recall  f1-score   support
#     business       1.00      1.00      1.00         1
#       sports       1.00      1.00      1.00         1

Step 6 — Predicting New, Unseen Text

new_headline = "Company releases new AI-powered laptop"
cleaned_new = preprocess(new_headline)
vector = vectorizer.transform([cleaned_new])
print(model.predict(vector))   # ['tech']

Wrapping It All in a scikit-learn Pipeline (Production Style)

from sklearn.pipeline import Pipeline

full_pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(preprocessor=preprocess)),
    ('classifier', MultinomialNB())
])

full_pipeline.fit(headlines, labels)   # raw text in, preprocessing handled internally
print(full_pipeline.predict(["Championship final ends in dramatic penalty shootout"]))
# ['sports']

This same pattern — preprocess → vectorize → train classifier → evaluate → predict — is the template for the next two applications we build: sentiment analysis and, using the same TF-IDF/BoW foundations, information extraction.