Correlation

Correlation measures the strength and direction of the linear relationship between two variables. It is the first bivariate technique an analyst reaches for.

Covariance — The Foundation

              Σ(x − x̄)(y − ȳ)
  Cov(x,y) = ──────────────────        (sample, divisor n−1)
                    n − 1

Covariance tells you the direction of the relationship (positive/negative), but its magnitude depends entirely on the units — covariance between height in cm and weight in kg is a different number from height in metres and weight in grams, for identical data. That unit-dependence is exactly what correlation fixes.

Karl Pearson's Correlation Coefficient (r)

            Cov(x, y)              Σ(x − x̄)(y − ȳ)
   r = ─────────────────  =  ────────────────────────────────
            σx · σy            √[Σ(x − x̄)²] · √[Σ(y − ȳ)²]

Computational (shortcut) form:

                 n·Σxy − Σx·Σy
   r = ───────────────────────────────────────────
        √[n·Σx² − (Σx)²] · √[n·Σy² − (Σy)²]

   Range:  −1 ≤ r ≤ +1     (unit-free)

Interpreting r

Worked Example — By Hand

Hours studied (x) vs marks (y):

n = 5

Numerator   = n·Σxy − Σx·Σy
            = 5(2080) − (30)(310)
            = 10400 − 9300 = 1100

Denominator = √[5(220) − 30²] × √[5(20450) − 310²]
            = √[1100 − 900] × √[102250 − 96100]
            = √200 × √6150
            = 14.142 × 78.422
            = 1108.99

r = 1100 / 1108.99 = 0.9919

Interpretation: r ≈ 0.99 — a very strong positive linear relationship. More study hours are associated with higher marks.

Coefficient of Determination (r²)

   r² = 0.9919² = 0.9839  ->  98.39%

r² is the proportion of variance in y explained by x. Here, 98.4% of the variation in marks is explained by variation in study hours; the remaining 1.6% comes from other factors.

Spearman's Rank Correlation (ρ)

Used for ordinal data, non-linear but monotonic relationships, or when outliers would distort Pearson's r.

              6 Σd²
   ρ = 1 − ─────────────       d = difference between the two ranks of an item
             n(n² − 1)                 n = number of pairs

Worked example. Ranks given by two judges:

ρ = 1 − (6 × 4) / (5 × (25 − 1))
  = 1 − 24 / 120
  = 1 − 0.2
  = 0.8    ->  strong agreement between the judges

Pearson vs Spearman vs Kendall

Python

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "hours": [2, 4, 6, 8, 10],
    "marks": [40, 50, 65, 70, 85],
})

print("Covariance:\n", df.cov().round(2))
#         hours   marks
# hours    10.0    27.5
# marks    27.5    77.5

print("Pearson r:", round(df["hours"].corr(df["marks"]), 4))       # 0.9919
print("Spearman:", round(df["hours"].corr(df["marks"], method="spearman"), 4))  # 1.0
print("Kendall: ", round(df["hours"].corr(df["marks"], method="kendall"), 4))   # 1.0
print("r squared:", round(df["hours"].corr(df["marks"]) ** 2, 4))  # 0.9839

# Correlation matrix + significance test
from scipy import stats

data = pd.DataFrame({
    "study_hours":  [2, 4, 6, 8, 10, 3, 7, 9],
    "marks":        [40, 50, 65, 70, 85, 45, 68, 80],
    "screen_time":  [8, 7, 5, 4, 2, 7, 4, 3],
    "attendance":   [60, 70, 80, 85, 95, 65, 82, 90],
})

print(data.corr().round(3))
#              study_hours  marks  screen_time  attendance
# study_hours        1.000  0.993       -0.980       0.988
# marks              0.993  1.000       -0.977       0.995
# screen_time       -0.980 -0.977        1.000      -0.973    <- NEGATIVE correlation
# attendance         0.988  0.995       -0.973       1.000

r, p_value = stats.pearsonr(data["study_hours"], data["marks"])
print(f"r = {r:.4f}, p-value = {p_value:.6f}")
# p < 0.05  ->  the correlation is statistically significant (see hypothesis testing lesson)

# Visualising the correlation matrix as a heatmap
import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(7, 5))
sns.heatmap(data.corr(), annot=True, cmap="coolwarm", center=0, fmt=".2f",
            square=True, linewidths=0.5)
plt.title("Correlation Heatmap")
plt.tight_layout()
plt.show()

Correlation Does NOT Imply Causation

The single most important caveat in all of analytics.

Why two variables can correlate without one causing the other:

Only a controlled, randomised experiment can establish causation — which is why A/B testing exists.

Anscombe's Quartet — Why You Must Also Plot

Four datasets with identical means, variances, correlations (r = 0.816) and regression lines, but completely different shapes: one linear, one curved, one linear-with-outlier, one dominated by a single point.

import seaborn as sns
df_ans = sns.load_dataset("anscombe")
print(df_ans.groupby("dataset").agg(
    x_mean=("x", "mean"), y_mean=("y", "mean"),
    corr=("x", lambda s: s.corr(df_ans.loc[s.index, "y"]))
).round(3))
# All four datasets: nearly identical statistics, wildly different scatter plots

Lesson: never report a correlation coefficient without looking at the scatter plot — the subject of a later lesson in this unit.