Data Preprocessing

Data preprocessing is the broader stage that converts raw data into a form suitable for analysis and modelling. Data cleaning is one component of it; the full set of tasks is usually stated as four (some books say five):

Why Preprocess?

1. Data Cleaning

Covered in the previous lesson — missing values, noise/outliers, duplicates, and inconsistencies.

Noise-smoothing techniques worth naming in an exam:

# Smoothing by BIN MEANS — classic textbook example
data = [4, 8, 15, 21, 21, 24, 25, 28, 34]
bin_size = 3
smoothed = []
for i in range(0, len(data), bin_size):
    chunk = data[i:i + bin_size]
    mean = sum(chunk) / len(chunk)
    smoothed.extend([round(mean, 1)] * len(chunk))

print("Original:", data)
print("Bin means:", smoothed)
# Original: [4, 8, 15, 21, 21, 24, 25, 28, 34]
# Bin means: [9.0, 9.0, 9.0, 22.0, 22.0, 22.0, 29.0, 29.0, 29.0]

# Smoothing by BIN BOUNDARIES — replace each value with the nearer boundary
smoothed_b = []
for i in range(0, len(data), bin_size):
    chunk = data[i:i + bin_size]
    lo, hi = min(chunk), max(chunk)
    smoothed_b.extend([lo if abs(v - lo) <= abs(v - hi) else hi for v in chunk])
print("Bin boundaries:", smoothed_b)
# Bin boundaries: [4, 4, 15, 21, 21, 24, 25, 25, 34]

2. Data Integration

Combining data from multiple sources — databases, files, APIs — into a single coherent store.

Problems to solve during integration:

Detecting redundancy — correlation analysis for numeric attributes, chi-square test for categorical ones. Highly correlated attributes carry duplicate information; keep one.

import pandas as pd

students = pd.DataFrame({"roll": [1, 2, 3], "name": ["Riya", "Zoya", "Kabir"]})
marks    = pd.DataFrame({"roll": [1, 2, 4], "marks": [88, 76, 65]})

inner = pd.merge(students, marks, on="roll", how="inner")   # only matching rolls
left  = pd.merge(students, marks, on="roll", how="left")    # all students
outer = pd.merge(students, marks, on="roll", how="outer")   # everything
print(left)
#    roll   name  marks
# 0     1   Riya   88.0
# 1     2   Zoya   76.0
# 2     3  Kabir    NaN     <- integration created a new missing value

# Redundancy check
df = pd.DataFrame({"monthly": [30000, 45000, 52000], "annual": [360000, 540000, 624000]})
print(df.corr().round(3))
# correlation = 1.000  ->  perfectly redundant, drop one column

3. Data Reduction

Producing a smaller representation of the dataset that yields (almost) the same analytical results.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

X = np.array([[2.5, 2.4, 1.1], [0.5, 0.7, 0.3], [2.2, 2.9, 1.5],
              [1.9, 2.2, 1.0], [3.1, 3.0, 1.6], [2.3, 2.7, 1.2]])

X_scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

print("Explained variance ratio:", pca.explained_variance_ratio_.round(3))
# e.g. [0.955 0.036]  ->  2 components retain ~99% of the information from 3 features
print("Reduced shape:", X_pca.shape)   # (6, 2)

# Aggregation — numerosity reduction on a time series
daily = pd.DataFrame({
    "date": pd.date_range("2026-01-01", periods=90),
    "sales": np.random.randint(800, 2200, 90),
})
monthly = daily.set_index("date").resample("ME")["sales"].sum()
print(monthly)      # 90 rows reduced to 3, trend preserved

4. Data Transformation

Converting data into forms appropriate for mining — normalization, standardization, encoding, aggregation, and attribute construction. Covered fully in the next lesson.

5. Data Discretization

Converting continuous attributes into a finite set of intervals/labels.

ages = pd.Series([12, 19, 23, 27, 34, 41, 45, 52, 63, 71])

# Equal-width: three bins of equal age range
print(pd.cut(ages, bins=3).value_counts().sort_index())

# Equal-frequency: three bins with equal counts
print(pd.qcut(ages, q=3).value_counts().sort_index())

# Custom, domain-driven bins with labels
groups = pd.cut(ages, bins=[0, 18, 35, 60, 100],
                labels=["Minor", "Young Adult", "Middle Aged", "Senior"])
print(pd.DataFrame({"age": ages, "group": groups}))

End-to-End Preprocessing Pipeline

import pandas as pd, numpy as np
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

df = pd.DataFrame({
    "age":     [19, np.nan, 21, 20, 22],
    "income":  [25000, 42000, np.nan, 31000, 58000],
    "city":    ["Delhi", "Noida", "Delhi", np.nan, "Noida"],
})

numeric_features = ["age", "income"]
categorical_features = ["city"]

numeric_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
categorical_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("encode", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipe, numeric_features),
    ("cat", categorical_pipe, categorical_features),
])

processed = preprocessor.fit_transform(df)
print(processed.shape)     # (5, 4)  ->  2 scaled numeric + 2 one-hot city columns

Building preprocessing as a pipeline rather than ad-hoc steps is what makes it reproducible — the same object can be re-applied to new data in production, guaranteeing training and inference see identical treatment.