Data Collection

Data collection is the systematic gathering of data relevant to the defined problem. It is the second step of the analytics process, and every downstream result inherits its quality — biased or incomplete collection cannot be repaired by clever modelling later.

Primary vs Secondary Data

Primary Data Collection Methods

Questionnaire Design — Key Rules

1. Keep questions short, simple and unambiguous.
2. Avoid leading questions — "Don't you agree our service is excellent?" ✗
3. Avoid double-barrelled questions — "Was the food tasty and affordable?" ✗ (two questions in one)
4. Use consistent scales (e.g. a 5-point Likert scale: Strongly Disagree → Strongly Agree).
5. Put sensitive/demographic questions at the end.
6. Mix closed-ended (easy to quantify) and a few open-ended (rich detail) questions.
7. Pilot test on a small group before full rollout.

Secondary Data Sources

Always evaluate secondary data before trusting it: Who collected it? When? For what purpose? What was the sample? What definitions were used? Has it been revised?

Data Collection in Practice — Python

import pandas as pd

# 1. From a CSV file (most common secondary source)
# df = pd.read_csv("students.csv")

# 2. From an Excel workbook
# df = pd.read_excel("sales.xlsx", sheet_name="Q1")

# 3. From a public API (JSON response)
import requests
# resp = requests.get("https://api.example.com/v1/records")
# df = pd.json_normalize(resp.json()["data"])

# 4. From a SQL database
# import sqlite3
# conn = sqlite3.connect("college.db")
# df = pd.read_sql_query("SELECT roll, name, marks FROM students", conn)

# 5. From a web page table (simple scraping)
# tables = pd.read_html("https://en.wikipedia.org/wiki/List_of_Indian_states")
# df = tables[0]

# 6. Manually entered / survey export
survey = pd.DataFrame({
    "respondent": [1, 2, 3, 4],
    "age":        [19, 21, 20, 22],
    "satisfaction": [4, 5, 3, 5],       # 5-point Likert
    "would_recommend": ["Yes", "Yes", "No", "Yes"],
})
print(survey.describe())

Sources of Error in Data Collection

Sampling error shrinks as the sample grows. Non-sampling error does not — a badly worded question asked of a million people is still badly worded. This is why collection design matters more than sheer volume.

Ethical and Legal Considerations

Once data is collected, the immediate question is how much and which subset to analyse — the topic of the next lesson, sampling.