p-Values: Evidence and Interpretation

1. Mathematical Definition of a p-Value

The p-value is the exact probability of obtaining a test statistic at least as extreme as the one observed, assuming the Null Hypothesis (H₀) is completely true.

Mathematically, it is the area under the tail(s) of the Probability Density Function (PDF) beyond the calculated test statistic.

• Right-Tailed Test: p = ∫[t_stat to ∞] f(x) dx
• Two-Tailed Test: p = 2 * ∫[|t_stat| to ∞] f(x) dx

2. The Relationship Between p-Value and α

The Significance Level (α) is an arbitrary threshold set by the researcher before the experiment (usually 0.05). The p-value is calculated from the data.

• If p ≤ α: The data is incompatible with H₀. We cross the threshold of "reasonable doubt" and Reject H₀.
• If p > α: The data could reasonably happen by chance under H₀. We Fail to Reject H₀.

3. The Anatomy of Statistical Significance

When a result is "Statistically Significant", it simply means: We are confident the true effect is not zero.

Crucial Distinction for Analysts:

• Statistical Significance (p-value): Does an effect exist? (Heavily influenced by sample size n).
• Practical Significance (Effect Size): Is the effect actually useful or large enough to care about in the real world?
• Example: With 1 million server logs, an algorithm that improves speed by 0.001 milliseconds might yield a p-value of 0.00001 (Highly Statistically Significant), but it has zero Practical Significance.

4. Severe Misconceptions