Regression
Regression is the supervised learning task of predicting a continuous numeric value from one or more input variables. Where classification answers "which class?", regression answers "how much?".
Terminology
| Term | Meaning |
|---|
| Dependent variable (Y) | What you predict — also target, response, outcome |
| Independent variable (X) | What you predict from — also predictor, feature, explanatory variable |
| Coefficient (β) | How much Y changes per one-unit change in X |
| Intercept (β₀) | Predicted Y when all X = 0 |
| Residual (e) | Actual Y − Predicted Ŷ, for one observation |
| Fitted value (Ŷ) | The model's prediction |
Classification vs Regression
| Basis | Classification | Regression |
|---|
| Output type | Discrete class label | Continuous number |
| Question | Which category? | How much / how many? |
| Example | Will the student pass? | What marks will the student score? |
| Algorithms | Naïve Bayes, KNN, decision tree, SVM | Linear, polynomial, ridge, lasso, SVR |
| Evaluation | Accuracy, precision, recall, F1 | MAE, MSE, RMSE, R² |
| Decision boundary | Separates classes | Best-fit line/surface through the points |
Logistic regression is a classification algorithm despite its name — it predicts a probability of class membership, then thresholds it. A frequent exam trap.
Regression Evaluation Metrics
1
MAE (Mean Absolute Error) = ─── Σ |yᵢ − ŷᵢ|
n
1
MSE (Mean Squared Error) = ─── Σ (yᵢ − ŷᵢ)²
n
RMSE (Root Mean Squared Error) = √MSE
SS_res Σ(yᵢ − ŷᵢ)²
R² (Coefficient of Determination) = 1 − ────── = 1 − ──────────────
SS_tot Σ(yᵢ − ȳ)²
(1 − R²)(n − 1)
Adjusted R² = 1 − ───────────────────────────────────
n − k − 1 (k = number of predictors)
1 | yᵢ − ŷᵢ |
MAPE (Mean Absolute % Error) = ─── Σ ────────── × 100
n yᵢ
| Metric | Unit | Outlier sensitivity | Interpretation |
|---|
| MAE | Same as Y | Low | "On average we're off by X units" |
| MSE | Y squared | High (squares the errors) | Used for optimisation, hard to interpret |
| RMSE | Same as Y | High | Most reported; penalises large errors |
| R² | Unitless (0–1) | Moderate | "% of variance in Y explained by the model" |
| Adjusted R² | Unitless | Moderate | R² penalised for adding useless predictors |
| MAPE | % | High | Comparable across different scales; breaks when Y ≈ 0 |
Why Adjusted R² exists: plain R² never decreases when you add a predictor, even a random one. Adjusted R² falls if the new predictor doesn't earn its place — so it is the honest metric for comparing models with different numbers of features.
Worked Example — Computing the Metrics
Actual: 10, 20, 30, 40 | Predicted: 12, 18, 33, 39
Errors (y − ŷ): −2, +2, −3, +1
MAE = (2 + 2 + 3 + 1) / 4 = 8/4 = 2.0
MSE = (4 + 4 + 9 + 1) / 4 = 18/4 = 4.5
RMSE = √4.5 = 2.121
ȳ = (10+20+30+40)/4 = 25
SS_tot = (10−25)² + (20−25)² + (30−25)² + (40−25)²
= 225 + 25 + 25 + 225 = 500
SS_res = 4 + 4 + 9 + 1 = 18
R² = 1 − 18/500 = 1 − 0.036 = 0.964 -> 96.4% of variance explained
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
y_true = np.array([10, 20, 30, 40])
y_pred = np.array([12, 18, 33, 39])
print(f"MAE = {mean_absolute_error(y_true, y_pred):.4f}") # 2.0000
print(f"MSE = {mean_squared_error(y_true, y_pred):.4f}") # 4.5000
print(f"RMSE = {np.sqrt(mean_squared_error(y_true, y_pred)):.4f}") # 2.1213
print(f"R² = {r2_score(y_true, y_pred):.4f}") # 0.9640
mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
print(f"MAPE = {mape:.2f}%") # 9.79%
Assumptions of Linear Regression — "LINE"
| Assumption | Meaning | How to check | Fix if violated |
|---|
| L — Linearity | Relationship between X and Y is linear | Scatter plot, residual plot | Transform X or Y; polynomial terms |
| I — Independence | Residuals are independent of each other | Durbin-Watson test; plot residuals in order | Time-series models |
| N — Normality | Residuals are normally distributed | Q-Q plot, Shapiro-Wilk on residuals | Transform Y (log/Box-Cox) |
| E — Equal variance (homoscedasticity) | Residual spread is constant across fitted values | Residuals vs fitted plot (look for a funnel) | Log transform, weighted least squares |
| (plus) No multicollinearity | Predictors are not strongly correlated with each other | VIF > 10 signals a problem | Drop or combine correlated predictors |
# Checking assumptions with residual diagnostics
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
from sklearn.linear_model import LinearRegression
np.random.seed(42)
X = np.random.uniform(1, 50, 120).reshape(-1, 1)
y = 3.2 * X.ravel() + 15 + np.random.normal(0, 8, 120)
model = LinearRegression().fit(X, y)
fitted = model.predict(X)
resid = y - fitted
fig, axes = plt.subplots(1, 3, figsize=(16, 4))
axes[0].scatter(fitted, resid, alpha=0.6, color="#168B99")
axes[0].axhline(0, color="#ef4444", linestyle="--")
axes[0].set_title("Residuals vs Fitted\n(want: random cloud)")
axes[0].set_xlabel("Fitted"); axes[0].set_ylabel("Residual")
stats.probplot(resid, dist="norm", plot=axes[1])
axes[1].set_title("Q-Q Plot\n(want: points on the line)")
axes[2].hist(resid, bins=20, color="#10b981", edgecolor="white")
axes[2].set_title("Residual Distribution\n(want: bell shaped, centred at 0)")
plt.tight_layout(); plt.show()
stat, p = stats.shapiro(resid)
print(f"Shapiro-Wilk on residuals: p = {p:.4f}",
"-> normality OK" if p > 0.05 else "-> normality violated")
# Multicollinearity check with VIF
from statsmodels.stats.outliers_influence import variance_inflation_factor
df = pd.DataFrame({
"area_sqft": [1000, 1500, 1200, 1800, 2000, 1350],
"area_sqm": [92.9, 139.4, 111.5, 167.2, 185.8, 125.4], # perfectly redundant!
"bedrooms": [2, 3, 2, 4, 4, 3],
})
vif = pd.DataFrame({
"feature": df.columns,
"VIF": [variance_inflation_factor(df.values, i) for i in range(df.shape[1])],
})
print(vif)
# area_sqft and area_sqm have astronomically high VIF — they are the same variable
# in different units. Drop one.
Choosing a Regression Model
The next lessons apply these ideas concretely: linear regression in full detail, plus the two classifiers named in the syllabus.
