Classification

Classification is the supervised learning task of predicting a categorical class label for a new observation, using a model learned from labelled training data.

Types of Classification

Major Classification Algorithms

Evaluating a Classifier — The Confusion Matrix

                       PREDICTED
                    Positive    Negative
                 ┌───────────┬───────────┐
        Positive │    TP     │    FN     │   <- actual positives
 ACTUAL          │(hit)      │(miss,     │
                 │           │ Type II)  │
                 ├───────────┼───────────┤
        Negative │    FP     │    TN     │   <- actual negatives
                 │(false     │(correct   │
                 │ alarm,    │ rejection)│
                 │ Type I)   │           │
                 └───────────┴───────────┘

Evaluation Metrics

                       TP + TN
   Accuracy   = ───────────────────────      overall correctness
                 TP + TN + FP + FN

                     TP
   Precision  = ───────────      of everything predicted positive, how much was right?
                 TP + FP

                     TP
   Recall     = ───────────      of all actual positives, how many did we catch?
   (Sensitivity) TP + FN

                     TN
   Specificity = ───────────     of all actual negatives, how many did we correctly reject?
                  TN + FP

                  2 × Precision × Recall
   F1-Score   = ──────────────────────────    harmonic mean of precision and recall
                   Precision + Recall

Worked Example — Fraud Detection

Out of 1,000 transactions, 50 are fraudulent. A model predicts 40 as fraud, of which 30 are genuinely fraudulent.

TP = 30                        (fraud correctly caught)
FP = 40 − 30 = 10              (legitimate wrongly flagged)
FN = 50 − 30 = 20              (fraud missed)
TN = 1000 − 30 − 10 − 20 = 940

Accuracy    = (30 + 940) / 1000        = 0.970  ->  97%
Precision   = 30 / (30 + 10) = 30/40   = 0.750  ->  75%
Recall      = 30 / (30 + 20) = 30/50   = 0.600  ->  60%
Specificity = 940 / (940 + 10)         = 0.989  ->  98.9%
F1          = 2(0.75 × 0.60)/(0.75 + 0.60) = 0.90/1.35 = 0.667

Interpretation: 97% accuracy sounds excellent, but the model misses 40% of all fraud (recall = 0.60). This is the accuracy paradox — on imbalanced data, always predicting "not fraud" would score 95% accuracy while catching nothing.

Precision vs Recall — Which Matters More?

There is always a trade-off: lowering the decision threshold catches more positives (↑ recall) but flags more negatives wrongly (↓ precision).

ROC Curve and AUC

ROC curve: plot True Positive Rate (Recall) against False Positive Rate (1 − Specificity)
           at every possible classification threshold.

AUC (Area Under Curve):
   1.0  = perfect classifier
   0.9+ = excellent
   0.8  = good
   0.7  = fair
   0.5  = no better than random guessing (the diagonal line)

Python — A Complete Classification Workflow

import pandas as pd
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, accuracy_score, precision_score,
                             recall_score, f1_score, classification_report, roc_auc_score)

data = load_breast_cancer()
X, y = data.data, data.target      # 0 = malignant, 1 = benign

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y   # stratify keeps class balance
)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)      # fit ONLY on training data
X_test_s = scaler.transform(X_test)            # transform test with the same scaler

model = LogisticRegression(max_iter=5000)
model.fit(X_train_s, y_train)
y_pred = model.predict(X_test_s)
y_proba = model.predict_proba(X_test_s)[:, 1]

print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
# [[51  2]
#  [ 1 89]]

print(f"\nAccuracy : {accuracy_score(y_test, y_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall   : {recall_score(y_test, y_pred):.4f}")
print(f"F1-Score : {f1_score(y_test, y_pred):.4f}")
print(f"ROC-AUC  : {roc_auc_score(y_test, y_proba):.4f}")

print("\n", classification_report(y_test, y_pred, target_names=data.target_names))

# Visualising the confusion matrix and ROC curve
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import roc_curve

fig, axes = plt.subplots(1, 2, figsize=(13, 5))

cm = confusion_matrix(y_test, y_pred)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", ax=axes[0],
            xticklabels=data.target_names, yticklabels=data.target_names)
axes[0].set_xlabel("Predicted"); axes[0].set_ylabel("Actual")
axes[0].set_title("Confusion Matrix")

fpr, tpr, thresholds = roc_curve(y_test, y_proba)
axes[1].plot(fpr, tpr, color="#168B99", linewidth=2,
             label=f"AUC = {roc_auc_score(y_test, y_proba):.3f}")
axes[1].plot([0, 1], [0, 1], "k--", label="Random (AUC = 0.5)")
axes[1].set_xlabel("False Positive Rate"); axes[1].set_ylabel("True Positive Rate")
axes[1].set_title("ROC Curve"); axes[1].legend()

plt.tight_layout(); plt.show()

# Comparing several classifiers on the same data
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

models = {
    "Naive Bayes":        GaussianNB(),
    "KNN (k=5)":          KNeighborsClassifier(n_neighbors=5),
    "Decision Tree":      DecisionTreeClassifier(random_state=42),
    "Random Forest":      RandomForestClassifier(n_estimators=100, random_state=42),
    "Logistic Regression": LogisticRegression(max_iter=5000),
}

results = []
for name, m in models.items():
    m.fit(X_train_s, y_train)
    pred = m.predict(X_test_s)
    results.append({
        "Model": name,
        "Accuracy": round(accuracy_score(y_test, pred), 4),
        "Precision": round(precision_score(y_test, pred), 4),
        "Recall": round(recall_score(y_test, pred), 4),
        "F1": round(f1_score(y_test, pred), 4),
    })

print(pd.DataFrame(results).sort_values("F1", ascending=False).to_string(index=False))

Handling Imbalanced Classes

The next lessons cover the two specific classifiers in this syllabus — Naïve Bayes and KNN — but first, the other supervised task: regression.