Data Transformation

Data transformation converts data from one format, scale or structure into another so that algorithms can use it effectively. It is the final preparation step before analysis.

1. Normalization and Standardization

The most examined part of this topic. When features live on wildly different scales, distance- and gradient-based algorithms (KNN, K-Means, SVM, neural networks, PCA) are dominated by the large-scale feature.

Min-Max Normalization

                 v - min(A)
    v' = ─────────────────────── × (new_max - new_min) + new_min
              max(A) - min(A)

For the common [0, 1] range this simplifies to:

                 v - min(A)
    v' = ───────────────────────
              max(A) - min(A)

Worked example. Income ranges from ₹12,000 to ₹98,000. Normalize ₹73,600 to [0, 1]:

v' = (73600 - 12000) / (98000 - 12000)
   = 61600 / 86000
   = 0.7163

To map it to the range [0, 1] is the default; for [-1, 1] you would multiply by 2 and subtract 1.

Z-Score Standardization

             v - μ            where μ = mean of attribute A
    v' = ───────────                σ = standard deviation of A
               σ

Result: mean = 0, standard deviation = 1

Worked example. Marks have μ = 65, σ = 12. Standardize a score of 83:

z = (83 - 65) / 12 = 18 / 12 = 1.5
=> the student is 1.5 standard deviations above the mean

Decimal Scaling

            v
    v' = ───────      where j is the smallest integer such that max(|v'|) < 1
          10^j

Example: values range up to 917  ->  j = 3  ->  917 / 1000 = 0.917

Robust Scaling

           v - median(A)
    v' = ─────────────────        (uses IQR instead of σ)
              IQR(A)

Best choice when outliers are present but must be retained.

Comparison Table

import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler, StandardScaler, RobustScaler

df = pd.DataFrame({
    "age":    [19, 21, 20, 22, 45],
    "income": [25000, 42000, 31000, 58000, 950000],   # note the outlier
})

print("--- Min-Max ---")
print(pd.DataFrame(MinMaxScaler().fit_transform(df), columns=df.columns).round(3))
# the outlier squashes every other income near 0

print("--- Z-score ---")
print(pd.DataFrame(StandardScaler().fit_transform(df), columns=df.columns).round(3))

print("--- Robust ---")
print(pd.DataFrame(RobustScaler().fit_transform(df), columns=df.columns).round(3))
# median/IQR based -> the four normal incomes stay well spread out

# Manual min-max, exactly as computed in an exam
income = df["income"]
manual = (income - income.min()) / (income.max() - income.min())
print(manual.round(4).tolist())
# [0.0, 0.0184, 0.0065, 0.0357, 1.0]

Critical rule: fit the scaler on the training set only, then apply it to the test set. Fitting on the whole dataset leaks test information into training.

2. Encoding Categorical Variables

Most algorithms accept only numbers, so categorical text must be encoded.

df = pd.DataFrame({
    "city":  ["Delhi", "Noida", "Delhi", "Gurgaon"],
    "size":  ["Small", "Large", "Medium", "Large"],
})

# ONE-HOT for nominal 'city'
print(pd.get_dummies(df["city"], prefix="city").astype(int))
#    city_Delhi  city_Gurgaon  city_Noida
# 0           1             0           0
# 1           0             0           1
# 2           1             0           0
# 3           0             1           0

# DUMMY encoding (k-1 columns) — drops the first level
print(pd.get_dummies(df["city"], prefix="city", drop_first=True).astype(int))

# ORDINAL encoding for a genuinely ordered attribute
order = {"Small": 0, "Medium": 1, "Large": 2}
df["size_encoded"] = df["size"].map(order)
print(df[["size", "size_encoded"]])

Why not label-encode 'city'? Encoding Delhi=0, Gurgaon=1, Noida=2 tells a distance-based model that Noida is "twice as far" from Delhi as Gurgaon is — a relationship that does not exist. Nominal ⇒ one-hot.

3. Functional (Mathematical) Transformations

Used to reduce skewness and stabilise variance so that models assuming normality behave better.

import numpy as np

income = pd.Series([25000, 30000, 32000, 41000, 55000, 90000, 250000, 900000])
print("Skewness before:", round(income.skew(), 3))          # 2.556  (heavily right-skewed)

log_income = np.log1p(income)                               # log(1 + x) handles zeros
print("Skewness after log:", round(log_income.skew(), 3))   # 1.033  (much closer to normal)

sqrt_income = np.sqrt(income)
print("Skewness after sqrt:", round(sqrt_income.skew(), 3)) # 1.845

4. Aggregation

Combining two or more attributes/records into a single summary — daily → monthly, transaction-level → customer-level.

sales = pd.DataFrame({
    "customer": ["C1", "C1", "C2", "C2", "C2", "C3"],
    "amount":   [1200, 800, 450, 900, 1100, 3000],
})

customer_level = sales.groupby("customer")["amount"].agg(
    total="sum", average="mean", orders="count", largest="max"
).round(2)
print(customer_level)
#           total   average  orders  largest
# customer
# C1         2000   1000.00       2     1200
# C2         2450    816.67       3     1100
# C3         3000   3000.00       1     3000

5. Attribute Construction (Feature Engineering)

Creating new attributes from existing ones — often the highest-impact step in the whole pipeline.

df = pd.DataFrame({
    "height_cm": [165, 172, 158, 180],
    "weight_kg": [60, 78, 52, 92],
    "dob": pd.to_datetime(["2005-04-12", "2003-11-30", "2006-01-08", "2002-07-19"]),
    "order_date": pd.to_datetime(["2026-01-14", "2026-02-01", "2026-03-09", "2026-03-15"]),
})

# Derived ratio feature
df["bmi"] = (df["weight_kg"] / (df["height_cm"] / 100) ** 2).round(2)

# Age from date of birth
df["age"] = ((pd.Timestamp("2026-08-07") - df["dob"]).dt.days // 365)

# Date parts — very common in real projects
df["order_month"] = df["order_date"].dt.month
df["order_weekday"] = df["order_date"].dt.day_name()
df["is_weekend"] = df["order_date"].dt.dayofweek >= 5

print(df[["bmi", "age", "order_month", "order_weekday", "is_weekend"]])

6. Generalization (Concept Hierarchy)

Replacing low-level values with higher-level concepts:

Street        →  City       →  State           →  Country
Connaught Pl. →  Delhi      →  Delhi NCR       →  India

Age 19, 21, 23 →  "Young Adult"  →  "Adult"

Complete Transformation Summary

With Unit 1 complete, the data is now collected, sampled, cleaned, integrated, reduced and transformed — an analysis-ready dataset. Unit 2 begins the actual analysis: summarising it with statistics and exploring it visually.