Data Quality: Missing Values & Outliers

1. Missing Values

Handling missing data is critical because most statistical methods and machine learning algorithms cannot process incomplete inputs.

Study Deep: MCAR, MAR, and MNAR

Understanding why data is missing is more important than knowing how to fill it.

1. MCAR (Completely at Random): No pattern. (e.g., A sensor battery died). Solution: Delete or Mean Impute.
2. MAR (At Random): Pattern depends on other data. (e.g., Younger users skip the 'Annual Income' field). Solution: Impute using Age.
3. MNAR (Not at Random): Pattern depends on the missing value itself. (e.g., Users with very high income skip the field to protect privacy). Solution: Requires advanced domain modeling.

1. Missing Values

Treatment Methods — Decision Guide:

2. Outliers

Formal Definition: An outlier is a data observation that lies at an abnormal distance from other values in the sample. Statistically, it is a point that falls outside the expected range of the data distribution.

Example: Salaries: [40k, 42k, 45k, 1M, 43k] -> 1M is an outlier.

Types of Outliers:

• Point Outlier: A single data point far from the rest (e.g., Age = 200 in a human dataset).
• Contextual Outlier: Abnormal in a specific context (e.g., 30°C is normal in summer, outlier in winter).
• Collective Outlier: A subset of data points that is anomalous as a group (e.g., sudden traffic spike on a server).

Detection Methods:

Worked Example (IQR Method): Data: [10, 15, 18, 20, 22, 25, 100]

• Q1 (25th percentile) = 15
• Q3 (75th percentile) = 25
• IQR = 25 - 15 = 10
• Lower Bound = 15 - 1.5(10) = 0
• Upper Bound = 25 + 1.5(10) = 40
• 100 > 40 → 100 is an outlier.

Treatment Decision Framework:

3. Python Code: Handling Missing Values

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Age':    [25, 30, np.nan, 22, 35, np.nan],
    'Salary': [50000, 60000, 45000, np.nan, 70000, 55000],
    'Gender': ['M', 'F', 'M', np.nan, 'F', 'M']
})

# 1. Detect missing values
print(df.isnull().sum())           # Count per column
print(df.isnull().sum() / len(df)) # Percentage per column

# 2. Drop rows where > 50% values are missing
df_clean = df.dropna(thresh=len(df.columns) // 2)

# 3. Mean imputation (numeric columns)
df['Age'].fillna(df['Age'].mean(), inplace=True)

# 4. Median imputation (robust to outliers)
df['Salary'].fillna(df['Salary'].median(), inplace=True)

# 5. Mode imputation (categorical columns)
df['Gender'].fillna(df['Gender'].mode()[0], inplace=True)

4. Python Code: Outlier Detection with IQR

import pandas as pd

data = pd.Series([10, 15, 18, 20, 22, 25, 100])

# IQR Method
Q1 = data.quantile(0.25)  # 15
Q3 = data.quantile(0.75)  # 25
IQR = Q3 - Q1             # 10

lower = Q1 - 1.5 * IQR   # 0
upper = Q3 + 1.5 * IQR   # 40

outliers = data[(data < lower) | (data > upper)]
print("Outliers:", outliers.values)   # [100]

# Z-Score Method
from scipy import stats
z_scores = stats.zscore(data)
outliers_z = data[abs(z_scores) > 3]
print("Z-Score Outliers:", outliers_z.values)

# Capping / Winsorization (replace outliers at threshold)
data_capped = data.clip(lower=lower, upper=upper)
print("After Capping:", data_capped.values)  # 100 → 40

5. Exam-Ready Summary