Probability Basics

Probability quantifies uncertainty — how likely an event is to occur. It is the mathematical foundation of hypothesis testing (later in this unit), Naïve Bayes classification (Unit 3), and every statistical inference an analyst makes.

Fundamental Terminology

Classical Definition

              Number of favourable outcomes        n(E)
   P(E) = ─────────────────────────────────── = ────────
              Total number of outcomes             n(S)

   0 ≤ P(E) ≤ 1
   P(impossible event) = 0
   P(certain event) = 1
   P(E) + P(E′) = 1

Three Approaches to Probability

Rules of Probability

Addition Rule (OR)

General:              P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
Mutually exclusive:   P(A ∪ B) = P(A) + P(B)          [since P(A ∩ B) = 0]

Worked example. A card is drawn from a standard 52-card deck. P(King or Heart)?

P(King)  = 4/52
P(Heart) = 13/52
P(King ∩ Heart) = 1/52          (the King of Hearts)

P(King ∪ Heart) = 4/52 + 13/52 − 1/52 = 16/52 = 4/13 ≈ 0.3077

Multiplication Rule (AND)

General:       P(A ∩ B) = P(A) × P(B | A)
Independent:   P(A ∩ B) = P(A) × P(B)

Worked example. Two cards drawn without replacement. P(both Kings)?

P(1st King) = 4/52
P(2nd King | 1st King) = 3/51          (dependent — one King already removed)
P(both) = (4/52) × (3/51) = 12/2652 = 1/221 ≈ 0.0045

With replacement they'd be independent: (4/52) × (4/52) = 1/169 ≈ 0.0059.

Conditional Probability

               P(A ∩ B)
   P(A | B) = ──────────        provided P(B) > 0
                 P(B)

"The probability of A given that B has already occurred."

Bayes' Theorem

The single most important formula for a data analyst — it reverses the direction of a conditional probability.

                P(B | A) × P(A)
   P(A | B) = ────────────────────
                     P(B)

Expanded with the law of total probability:

                     P(B | A) × P(A)
   P(A | B) = ─────────────────────────────────
               P(B | A)·P(A) + P(B | A′)·P(A′)

   P(A)     = prior probability
   P(B | A) = likelihood
   P(A | B) = posterior probability
   P(B)     = evidence / marginal probability

Classic Worked Example — Medical Test

A disease affects 1% of the population. A test is 95% accurate for those who have it (sensitivity) and gives a 10% false-positive rate for those who don't. A person tests positive. What is the probability they actually have the disease?

Let D  = has disease,    D′ = no disease
Let +  = tests positive

P(D)      = 0.01          P(D′)      = 0.99
P(+ | D)  = 0.95          P(+ | D′)  = 0.10

P(+) = P(+|D)·P(D) + P(+|D′)·P(D′)
     = (0.95)(0.01) + (0.10)(0.99)
     = 0.0095 + 0.099
     = 0.1085

              P(+|D) · P(D)      0.0095
P(D | +) = ─────────────────── = ──────── = 0.0876
                   P(+)           0.1085

=> Only about 8.76% !

Why so low? Because the disease is rare, the 10% false positives from the huge healthy population (99 out of every 10,000 people) vastly outnumber the true positives (9.5 per 10,000). This is the base rate fallacy — and it is exactly why screening tests are re-run before diagnosis.

def bayes(prior, sensitivity, false_positive_rate):
    """P(disease | positive test)"""
    p_pos = sensitivity * prior + false_positive_rate * (1 - prior)
    return (sensitivity * prior) / p_pos

print(f"{bayes(0.01, 0.95, 0.10):.4f}")     # 0.0876

# Test again after a first positive — the posterior becomes the new prior
second = bayes(bayes(0.01, 0.95, 0.10), 0.95, 0.10)
print(f"After a second positive test: {second:.4f}")   # 0.4772
# Two independent positive tests raise it from 8.8% to 47.7%

Random Variables

A random variable assigns a numerical value to each outcome of a random experiment.

Expected value (mean):    E(X) = Σ x·P(x)          [discrete]
Variance:                 Var(X) = E(X²) − [E(X)]²

Worked example. A game: pay ₹10 to roll a die; win ₹60 if you roll a 6, otherwise nothing.

E(winnings) = 60 × (1/6) + 0 × (5/6) = 10
E(profit)   = 10 − 10 = 0        ->  a perfectly fair game

If the prize were ₹50:  E(winnings) = 50/6 = 8.33  ->  expected loss of ₹1.67 per play

import numpy as np

# Simulating probability empirically — the Law of Large Numbers in action
np.random.seed(42)
for n in [10, 100, 1000, 100000]:
    tosses = np.random.choice(["H", "T"], size=n)
    p_head = (tosses == "H").mean()
    print(f"n = {n:>6}:  P(head) = {p_head:.4f}")
# n =     10:  P(head) = 0.3000
# n =    100:  P(head) = 0.5100
# n =   1000:  P(head) = 0.4880
# n = 100000:  P(head) = 0.4992     -> converges to the theoretical 0.5

# Expected value of a discrete random variable
values = np.array([0, 1, 2, 3])            # number of heads in 3 tosses
probs  = np.array([1/8, 3/8, 3/8, 1/8])

ev = (values * probs).sum()
var = ((values ** 2) * probs).sum() - ev ** 2
print(f"E(X) = {ev:.2f}, Var(X) = {var:.2f}, SD = {var ** 0.5:.3f}")
# E(X) = 1.50, Var(X) = 0.75, SD = 0.866

Probability gives us the language of uncertainty. The next lesson turns that language into the specific distributions analysts encounter constantly.