Naïve Bayes Classifier

Naïve Bayes is a probabilistic classifier based on Bayes' theorem with a strong ("naïve") assumption that all features are conditionally independent given the class. Despite that assumption being almost never literally true, it performs remarkably well — especially on text.

The Mathematics

                    P(X | C) · P(C)
   P(C | X) = ─────────────────────────
                        P(X)

   C = class,  X = feature vector (x₁, x₂, …, xₙ)

The NAIVE assumption — features are conditionally independent given C:

   P(X | C) = P(x₁ | C) × P(x₂ | C) × … × P(xₙ | C)

So:
   P(C | X) ∝ P(C) × ∏ P(xᵢ | C)

Prediction: choose the class with the highest posterior probability

   ĉ = argmax_C  P(C) × ∏ P(xᵢ | C)

(P(X) is dropped — it is the same constant for every class, so it cannot
change which class wins.)

Worked Example — The Classic "Play Tennis" Problem

Question: Should we play on a day that is Sunny, Cool, High humidity, Windy = True?

Step 1 — Prior probabilities

Total days = 14,  Yes = 9,  No = 5

P(Yes) = 9/14 = 0.643
P(No)  = 5/14 = 0.357

Step 2 — Likelihoods (conditional probabilities)

For class YES (9 days):
   P(Sunny | Yes)    = 2/9 = 0.222
   P(Cool | Yes)     = 3/9 = 0.333
   P(High | Yes)     = 3/9 = 0.333
   P(Windy=T | Yes)  = 3/9 = 0.333

For class NO (5 days):
   P(Sunny | No)     = 3/5 = 0.600
   P(Cool | No)      = 1/5 = 0.200
   P(High | No)      = 4/5 = 0.800
   P(Windy=T | No)   = 3/5 = 0.600

Step 3 — Posterior (unnormalised)

P(Yes | X) ∝ 0.643 × 0.222 × 0.333 × 0.333 × 0.333
           = 0.005274

P(No | X)  ∝ 0.357 × 0.600 × 0.200 × 0.800 × 0.600
           = 0.020563

Step 4 — Normalise and decide

Total = 0.005274 + 0.020563 = 0.025837

P(Yes | X) = 0.005274 / 0.025837 = 0.204   ->  20.4%
P(No  | X) = 0.020563 / 0.025837 = 0.796   ->  79.6%

DECISION: NO — do not play tennis  (79.6% confidence)

The Zero-Frequency Problem and Laplace Smoothing

If a feature value never appears with a class in the training data, its likelihood is 0 — and because the terms are multiplied, the entire posterior collapses to zero, no matter how strongly every other feature supports that class.

Laplace (add-one) smoothing:

                     count(xᵢ, C) + α
   P(xᵢ | C) = ───────────────────────────────
                  count(C) + α × k

   α = smoothing parameter (usually 1)
   k = number of distinct values the feature can take

Example: P(Overcast | No) = 0/5 = 0  ->  with α=1, k=3 outlook values:
                          = (0 + 1) / (5 + 3) = 1/8 = 0.125

Types of Naïve Bayes

For Gaussian NB, the likelihood uses the normal PDF with the class-wise mean and variance:

                      1              −(x − μ_c)²
   P(x | C) = ───────────────── · exp( ─────────── )
                √(2π σ_c²)               2σ_c²

Python

import pandas as pd
import numpy as np
from sklearn.naive_bayes import CategoricalNB
from sklearn.preprocessing import OrdinalEncoder

data = pd.DataFrame({
    "outlook":  ["Sunny","Sunny","Overcast","Rainy","Rainy","Rainy","Overcast",
                 "Sunny","Sunny","Rainy","Sunny","Overcast","Overcast","Rainy"],
    "temp":     ["Hot","Hot","Hot","Mild","Cool","Cool","Cool",
                 "Mild","Cool","Mild","Mild","Mild","Hot","Mild"],
    "humidity": ["High","High","High","High","Normal","Normal","Normal",
                 "High","Normal","Normal","Normal","High","Normal","High"],
    "windy":    [False,True,False,False,False,True,True,
                 False,False,False,True,True,False,True],
    "play":     ["No","No","Yes","Yes","Yes","No","Yes",
                 "No","Yes","Yes","Yes","Yes","Yes","No"],
})

X = data.drop(columns="play")
y = data["play"]

encoder = OrdinalEncoder()
X_enc = encoder.fit_transform(X)

model = CategoricalNB(alpha=1e-10)      # tiny alpha ≈ no smoothing, to match hand calc
model.fit(X_enc, y)

# Predict for: Sunny, Cool, High, Windy=True
new = pd.DataFrame([["Sunny", "Cool", "High", True]], columns=X.columns)
new_enc = encoder.transform(new)

print("Prediction:", model.predict(new_enc)[0])
print("Class order:", model.classes_)
print("Probabilities:", model.predict_proba(new_enc).round(4))
# Prediction: No
# Class order: ['No' 'Yes']
# Probabilities: [[0.7958 0.2042]]     <- matches the hand calculation

# GAUSSIAN Naive Bayes on continuous features
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=42, stratify=iris.target
)

gnb = GaussianNB()
gnb.fit(X_train, y_train)
pred = gnb.predict(X_test)

print(f"Accuracy: {accuracy_score(y_test, pred):.4f}")     # 0.9778
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, target_names=iris.target_names))

# The learned per-class parameters
print("\nClass priors:", gnb.class_prior_.round(3))
print("Class means (theta):\n", pd.DataFrame(gnb.theta_.round(2),
      index=iris.target_names, columns=iris.feature_names))

# MULTINOMIAL Naive Bayes for TEXT classification — the flagship use case
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

texts = [
    "win a free lottery prize now",       "claim your free cash prize",
    "urgent win money click here",        "free offer limited time click",
    "meeting scheduled for monday",       "please review the attached report",
    "lunch at one o clock today",         "project deadline moved to friday",
]
labels = ["spam"]*4 + ["ham"]*4

clf = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))
clf.fit(texts, labels)

tests = ["free money click now", "the report is ready for the meeting"]
for t in tests:
    pred = clf.predict([t])[0]
    proba = clf.predict_proba([t]).max()
    print(f"{t!r:45} -> {pred}  ({proba:.1%} confident)")
# 'free money click now'                     -> spam  (97.8% confident)
# 'the report is ready for the meeting'      -> ham   (96.4% confident)

Advantages and Disadvantages

Applications

Naïve Bayes learns a probabilistic model. The next algorithm, KNN, learns nothing at all until prediction time.