Naïve Bayes for Text Classification

Naïve Bayes is a probabilistic classifier based on Bayes' Theorem, with a "naïve" assumption that all features (words) are conditionally independent given the class. Despite this simplifying (and technically incorrect) assumption, it performs remarkably well for text classification.

Bayes' Theorem

P(class | document) = P(document | class) · P(class) / P(document)

For classification we only need to compare classes, so we can drop the constant denominator P(document):

class* = argmax_c  P(c) · Π P(w_i | c)      for each word w_i in the document

• P(c) — prior probability of the class (how common is "spam" vs "not spam" overall?)
• P(w_i | c) — likelihood of word w_i given class c, estimated from training data (this is the "naïve" independence assumption — each word's probability is computed independently)

Worked Example — Spam Classification

Training data:
  Spam:     "win money now", "win free prize"
  Not Spam: "meeting at office", "project deadline now"

P(Spam) = 2/4 = 0.5
P(Not Spam) = 2/4 = 0.5

Classify: "win prize now"
P(Spam | doc)     ∝ P(Spam) × P(win|Spam) × P(prize|Spam) × P(now|Spam)
P(Not Spam | doc) ∝ P(NotSpam) × P(win|NotSpam) × P(prize|NotSpam) × P(now|NotSpam)

Since "win" and "prize" appear only in Spam training examples,
P(Spam | doc) will be much higher -> classified as SPAM

Implementing Naïve Bayes with scikit-learn

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

# Training data
texts = [
    "win money now", "win free prize", "claim your prize now",
    "meeting at office", "project deadline now", "office lunch today"
]
labels = ["spam", "spam", "spam", "not spam", "not spam", "not spam"]

vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(texts)

model = MultinomialNB()
model.fit(X_train, labels)

# Predict a new message
test = ["free money offer"]
X_test = vectorizer.transform(test)
print(model.predict(X_test))          # ['spam']
print(model.predict_proba(X_test))    # [[P(not spam), P(spam)]]

Why "Multinomial" Naïve Bayes for Text?

MultinomialNB models word counts (how many times each word appears) — the natural fit for Bag-of-Words / TF-IDF features. Other variants exist:

Handling Zero Probabilities — Laplace Smoothing (Again!)

Just like the zero-probability problem in n-gram language models (Unit 2), if a word in the test document never appeared in training for a given class, its likelihood is 0 — zeroing the whole product. MultinomialNB applies the same Laplace (add-one) smoothing by default via its alpha parameter:

model = MultinomialNB(alpha=1.0)   # alpha=1.0 is standard Laplace smoothing

Naïve Bayes with TF-IDF Features

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ('tfidf', TfidfVectorizer()),
    ('classifier', MultinomialNB())
])

pipeline.fit(texts, labels)
print(pipeline.predict(["claim your free money now"]))   # ['spam']

Why Naïve Bayes Works Well Despite the "Naïve" Assumption

Even though words are rarely truly independent (e.g. "New" and "York" are highly correlated), Naïve Bayes only needs to get the relative ranking of classes right, not perfectly calibrated probabilities — which is a much easier bar to clear. Combined with its speed and small data requirements, this makes it a strong, fast baseline for text classification, spam detection, and (as we see next) sentiment analysis.