Introduction to Hypothesis Testing

Hypothesis testing is the formal statistical procedure for deciding whether sample evidence is strong enough to support a claim about a population. It converts "this difference looks real" into a defensible, quantified decision.

The Two Hypotheses

Examples:

Critical logic: we never "prove" or "accept" H₀. We either reject H₀ (evidence is strong) or fail to reject H₀ (evidence is insufficient). Failing to convict is not the same as proving innocence.

The Testing Procedure

Significance Level (α) and p-value

   Decision rule:

     p-value ≤ α   ->  REJECT H₀        (statistically significant)
     p-value >  α   ->  FAIL TO REJECT H₀

   Equivalent critical-value rule:
     |test statistic| > critical value  ->  REJECT H₀

What a p-value is NOT: it is not the probability that H₀ is true, and not the probability that your result occurred by chance. It is P(data this extreme | H₀ true).

Type I and Type II Errors

Memorable framing — a medical test:

• Type I error: telling a healthy person they have the disease (false alarm)
• Type II error: telling a sick person they are healthy (missed detection)

The trade-off: lowering α (e.g. 0.05 → 0.01) reduces Type I errors but increases Type II errors. The only way to reduce both simultaneously is to increase the sample size.

Statistical power = 1 − β — the probability of correctly detecting a real effect. Power increases with larger sample size, larger true effect size, smaller variance, and larger α.

One-Tailed vs Two-Tailed Tests

TWO-TAILED (H₁: μ ≠ μ₀)          α split between both tails: α/2 each
    Rejection regions:  Z < −1.96  or  Z > +1.96   for α = 0.05

RIGHT-TAILED (H₁: μ > μ₀)        entire α in the upper tail
    Rejection region:  Z > 1.645   for α = 0.05

LEFT-TAILED (H₁: μ < μ₀)         entire α in the lower tail
    Rejection region:  Z < −1.645  for α = 0.05

A one-tailed test has more power to detect an effect in the specified direction, but it cannot detect an effect in the opposite direction at all. Choose the direction before seeing the data.

Common Tests — Which One to Use

Test Reference Table

Worked Example — One-Sample Z-Test

A college claims its students' average score is 65 with σ = 12. A sample of 36 students has a mean of 69. Test at α = 0.05 whether the true mean is higher.

Step 1:  H₀: μ = 65        H₁: μ > 65      (right-tailed)
Step 2:  α = 0.05
Step 3:  σ known, n = 36 ≥ 30  ->  one-sample z-test

Step 4:  Standard Error = σ/√n = 12/√36 = 12/6 = 2

              x̄ − μ     69 − 65      4
         z = ───────── = ───────── = ─── = 2.0
              σ/√n           2         2

Step 5:  Critical value for right-tailed α = 0.05  ->  z_crit = 1.645
         p-value = P(Z > 2.0) = 1 − 0.9772 = 0.0228

Step 6:  z = 2.0 > 1.645   AND   p = 0.0228 < 0.05
         ->  REJECT H₀

Conclusion: There is sufficient evidence at the 5% significance level to
conclude that the true mean score is greater than 65.

Worked Example — One-Sample t-Test

A machine should fill 500 ml bottles. A sample of 10 bottles gives x̄ = 495 ml, s = 8 ml. Is the machine mis-calibrated? (α = 0.05)

Step 1:  H₀: μ = 500       H₁: μ ≠ 500     (two-tailed)
Step 2:  α = 0.05
Step 3:  σ unknown, n = 10 < 30  ->  one-sample t-test, df = 9

Step 4:  SE = s/√n = 8/√10 = 8/3.162 = 2.530

              495 − 500      −5
         t = ─────────── = ──────── = −1.976
                2.530        2.530

Step 5:  Two-tailed critical value, df = 9, α = 0.05  ->  t_crit = ±2.262
         p-value ≈ 0.0797

Step 6:  |−1.976| = 1.976 < 2.262   AND   p = 0.0797 > 0.05
         ->  FAIL TO REJECT H₀

Conclusion: There is insufficient evidence to conclude the machine is
mis-calibrated. (Note: with a larger sample this same 5 ml gap might well
become significant — "not significant" is not the same as "no effect".)

Python

import numpy as np
from scipy import stats

# ---- ONE-SAMPLE Z-TEST (computed manually; scipy has no built-in z-test) ----
x_bar, mu, sigma, n = 69, 65, 12, 36
z = (x_bar - mu) / (sigma / np.sqrt(n))
p_right = 1 - stats.norm.cdf(z)

print(f"z = {z:.4f}, p (right-tailed) = {p_right:.4f}")
print("Decision:", "Reject H0" if p_right < 0.05 else "Fail to reject H0")
# z = 2.0000, p (right-tailed) = 0.0228
# Decision: Reject H0

# ---- ONE-SAMPLE t-TEST ----
np.random.seed(42)
bottles = np.array([492, 498, 501, 489, 495, 503, 487, 496, 499, 490])

t_stat, p_val = stats.ttest_1samp(bottles, popmean=500)
print(f"Sample mean = {bottles.mean():.2f}")
print(f"t = {t_stat:.4f}, p = {p_val:.4f}")
print("Decision:", "Reject H0" if p_val < 0.05 else "Fail to reject H0")

# ---- TWO-SAMPLE INDEPENDENT t-TEST ----
section_a = np.array([72, 68, 75, 80, 71, 69, 77, 74, 66, 79])
section_b = np.array([65, 61, 70, 63, 68, 59, 66, 64, 62, 67])

t_stat, p_val = stats.ttest_ind(section_a, section_b)
print(f"Mean A = {section_a.mean():.2f}, Mean B = {section_b.mean():.2f}")
print(f"t = {t_stat:.4f}, p = {p_val:.6f}")
# Mean A = 73.10, Mean B = 64.50
# t = 4.5236, p = 0.000262   ->  reject H0: the sections genuinely differ

# ---- PAIRED t-TEST — before/after on the SAME students ----
before = np.array([65, 70, 58, 72, 61, 68, 74, 63])
after  = np.array([71, 74, 65, 75, 68, 73, 78, 70])

t_stat, p_val = stats.ttest_rel(before, after)
print(f"Mean improvement = {(after - before).mean():.2f} marks")
print(f"t = {t_stat:.4f}, p = {p_val:.6f}")
# Mean improvement = 5.62 marks
# p is very small  ->  the training programme had a significant effect

# ---- CHI-SQUARE TEST OF INDEPENDENCE ----
import pandas as pd

# Is course preference independent of gender?
observed = pd.DataFrame({
    "BCA": [30, 20],
    "BBA": [15, 25],
    "BSc": [25, 25],
}, index=["Male", "Female"])

chi2, p, dof, expected = stats.chi2_contingency(observed)
print("Observed:\n", observed)
print("\nExpected:\n", pd.DataFrame(expected.round(2),
      index=observed.index, columns=observed.columns))
print(f"\nchi-square = {chi2:.4f}, dof = {dof}, p = {p:.4f}")
print("Decision:", "Variables are DEPENDENT" if p < 0.05 else "Independent")

# ---- ANOVA — comparing THREE group means ----
sec_a = [72, 68, 75, 80, 71]
sec_b = [65, 61, 70, 63, 68]
sec_c = [78, 82, 76, 85, 80]

f_stat, p_val = stats.f_oneway(sec_a, sec_b, sec_c)
print(f"F = {f_stat:.4f}, p = {p_val:.6f}")
# p < 0.05 -> at least one section's mean differs
# (follow up with Tukey's HSD to find WHICH pairs differ)

Statistical vs Practical Significance

# Cohen's d — effect size
def cohens_d(a, b):
    n1, n2 = len(a), len(b)
    pooled_sd = np.sqrt(((n1-1)*np.var(a, ddof=1) + (n2-1)*np.var(b, ddof=1)) / (n1+n2-2))
    return (np.mean(a) - np.mean(b)) / pooled_sd

d = cohens_d(section_a, section_b)
print(f"Cohen's d = {d:.3f}")
# d = 2.023  ->  a very large effect (0.2 small, 0.5 medium, 0.8 large)

With a sample of 100,000, a difference of 0.01 marks can be "highly significant" (p < 0.001) and completely worthless. Always report the effect size alongside the p-value.

Assumptions and Common Mistakes

This concludes Unit 2. You can now describe data (central tendency, dispersion), relate variables (correlation), explore visually (EDA, histograms, box plots, scatter plots), and formally test claims. Unit 3 moves from describing data to learning from it — building predictive and descriptive models.