Confidence Intervals: Quantifying Uncertainty

1. Point Estimates vs. Interval Estimates

• Point Estimate: A single number calculated from a sample (e.g., sample mean x̄ = 45). It is almost guaranteed to be slightly wrong compared to the true population parameter.
• Interval Estimate (CI): A range of values built around the point estimate (e.g., [42.5, 47.5]). It incorporates the known sampling error to provide a realistic bound for the true parameter.

2. The Anatomy of a Confidence Interval

General Formula: Estimate ± (Critical Value Standard Error) The term (Critical Value Standard Error) is known as the Margin of Error (ME).

For a Population Mean (large sample): CI = x̄ ± Z * (s / √n)

• Z controls the Confidence Level (e.g., 1.96 for 95%, 2.576 for 99%).
• (s / √n) is the standard error.

3. Deep Interpretation of the 95% Level

A 95% Confidence Interval DOES NOT mean "there is a 95% probability the true mean is in this specific interval." (In frequentist statistics, the true mean is a fixed constant—it is either in the interval [100% prob] or it isn't [0% prob]).

Correct Interpretation: If we repeated this exact sampling process 100 times, creating 100 different intervals, we expect approximately 95 of those intervals to successfully contain the true population mean. We are 95% confident in our methodology.

4. Confidence Intervals for Proportions

Highly relevant in A/B testing (e.g., Conversion Rates). Formula: p̂ ± Z √((p̂ (1 - p̂)) / n)

• Where: p̂ is the sample proportion (e.g., 0.15 for 15% conversion).
• Insight: Maximum variance (widest interval) occurs when p̂ = 0.50.

5. Duality with Hypothesis Testing

A 95% Confidence Interval is mathematically equivalent to a two-tailed hypothesis test at α = 0.05.

• If the Null Hypothesis value (e.g., μ = 0 difference) falls outside the CI, the p-value will be < 0.05 (Reject H₀).
• If the CI contains the Null value, the p-value will be > 0.05 (Fail to Reject).
• Advantage of CI: It shows both statistical significance and the practical range of the effect size.