Big Data Fundamentals
Big Data refers to datasets so large, fast-moving or varied that traditional data-processing tools (a single machine, a relational database, Pandas in memory) cannot capture, store, manage or analyse them within an acceptable time.
Working definition: if your data comfortably fits in RAM on one machine and a Pandas script finishes in minutes, it is not big data — it is just data. Big data begins where single-machine processing breaks down.
The V's of Big Data
The Original 3 V's (Doug Laney, 2001)
| V | Meaning | Examples |
|---|
| Volume | Sheer quantity of data | Facebook stores 300+ PB; a single Boeing engine emits ~500 GB per flight |
| Velocity | Speed of generation and required processing | 6,000 tweets/second; stock ticks in microseconds; IoT sensor streams |
| Variety | Different formats and structures | Text, images, video, audio, logs, JSON, sensor readings, geospatial |
Extended V's
| V | Meaning | Why it matters |
|---|
| Veracity | Accuracy, trustworthiness, and uncertainty of the data | Dirty data at petabyte scale produces confidently wrong conclusions |
| Value | Ability to convert data into business benefit | Storage costs money; data with no value is a liability, not an asset |
| Variability | Inconsistency in meaning and flow over time | Sentiment of a word changes; traffic spikes during events |
| Visualization | Making massive data comprehensible | You cannot scatter-plot a billion points meaningfully |
Data Volume Scale
1 Kilobyte (KB) = 1,000 bytes ~ half a page of text
1 Megabyte (MB) = 1,000 KB ~ one MP3 song
1 Gigabyte (GB) = 1,000 MB ~ one HD movie
1 Terabyte (TB) = 1,000 GB ~ 250,000 photos
1 Petabyte (PB) = 1,000 TB ~ all US academic research libraries
1 Exabyte (EB) = 1,000 PB ~ all words ever spoken by humanity (est.)
1 Zettabyte (ZB) = 1,000 EB ~ global annual internet traffic
Types of Big Data
| Type | Share | Description | Examples |
|---|
| Structured | ~20% | Fixed schema, rows and columns | Transaction tables, sensor logs with fixed fields |
| Semi-structured | ~10% | Self-describing tags, flexible schema | JSON, XML, log files, emails |
| Unstructured | ~70–80% | No predefined model | Video, images, audio, free text, social media posts |
Traditional vs Big Data Systems
| Basis | Traditional (RDBMS) | Big Data Systems |
|---|
| Data volume | GB to a few TB | TB to PB and beyond |
| Data types | Structured only | All types |
| Schema | Schema-on-write — defined before loading | Schema-on-read — applied when queried |
| Architecture | Centralised, scale-up (bigger server) | Distributed, scale-out (more commodity servers) |
| Hardware | Expensive, specialised | Commodity, cheap, expendable |
| Processing | Batch, mostly interactive SQL | Batch + stream + interactive |
| Fault tolerance | Hardware redundancy (RAID, failover) | Software-level replication — assumes failure is normal |
| Data model | Relational, normalised, ACID | Often denormalised, NoSQL, BASE |
| Cost per TB | High | Low |
| Examples | Oracle, MySQL, PostgreSQL, SQL Server | Hadoop, Spark, Cassandra, MongoDB, Kafka |
Scale-Up vs Scale-Out
Big data architecture is built on scale-out. The insight is that a thousand cheap machines cost less than one supercomputer and can be added incrementally — as long as the software handles the inevitable failures.
The CAP Theorem
For any distributed data store, you can guarantee at most two of the following three:
C — CONSISTENCY: every read sees the most recent write
A — AVAILABILITY: every request receives a response
P — PARTITION TOLERANCE: the system keeps working despite network failures
Since network partitions are unavoidable in a distributed system, P is
mandatory — so the real design choice is between C and A.
| Choice | Description | Systems |
|---|
| CP | Consistent + partition tolerant; may reject requests during a partition | HBase, MongoDB (default), Redis |
| AP | Available + partition tolerant; may return stale data | Cassandra, DynamoDB, CouchDB |
| CA | Only possible in a single-node (non-distributed) system | Traditional RDBMS |
ACID vs BASE
| ACID (traditional RDBMS) | BASE (big data / NoSQL) |
|---|
| Atomicity — all or nothing | BAsically Available — responds even if degraded |
| Consistency — valid state always | Soft state — state may change without input |
| Isolation — concurrent transactions don't interfere | Eventual consistency — converges given enough time |
| Durability — committed data survives failures | |
NoSQL Databases
| Type | Data model | Best for | Examples |
|---|
| Key-Value | Simple key → value pairs | Caching, sessions, very fast lookups | Redis, DynamoDB, Riak |
| Document | JSON/BSON documents | Semi-structured content, flexible schemas | MongoDB, CouchDB |
| Column-family | Columns grouped into families | Time series, huge sparse tables | Cassandra, HBase |
| Graph | Nodes and edges | Relationships, social networks, fraud rings | Neo4j, Amazon Neptune |
Big Data Processing Paradigms
| Paradigm | Description | Latency | Technologies |
|---|
| Batch processing | Process large volumes accumulated over time | Minutes to hours | Hadoop MapReduce, Spark |
| Stream processing | Process each event as it arrives | Milliseconds to seconds | Kafka Streams, Flink, Spark Streaming, Storm |
| Micro-batch | Tiny batches at short intervals — a compromise | Seconds | Spark Structured Streaming |
| Interactive / ad-hoc | Fast SQL queries over big data | Seconds | Hive, Presto/Trino, Impala, Drill |
Lambda vs Kappa Architecture
Lambda gives accuracy plus speed at the cost of maintaining two codebases. Kappa simplifies to one streaming pipeline and re-derives history by replaying the event log.
The Big Data Technology Landscape
| Layer | Purpose | Technologies |
|---|
| Ingestion | Get data in | Kafka, Flume, Sqoop, NiFi |
| Storage | Store at scale | HDFS, Amazon S3, Azure Data Lake, Google Cloud Storage |
| Processing | Compute | MapReduce, Spark, Flink, Tez |
| Query | SQL access | Hive, Presto/Trino, Impala, Drill |
| NoSQL | Operational stores | Cassandra, HBase, MongoDB |
| Resource management | Cluster scheduling | YARN, Mesos, Kubernetes |
| Coordination | Distributed consensus | ZooKeeper |
| Workflow | Pipeline orchestration | Airflow, Oozie, Luigi |
| ML at scale | Distributed learning | Spark MLlib, TensorFlow, H2O |
| Visualization | Dashboards | Tableau, Power BI, Superset, Grafana |
Apache Spark — The Modern Standard
Spark superseded MapReduce as the default big-data processing engine.
| Hadoop MapReduce | Apache Spark |
|---|
| Processing | Disk-based — writes to disk between stages | In-memory |
| Speed | Baseline | 10–100× faster for iterative workloads |
| Ease of use | Verbose Java | Concise APIs in Python (PySpark), Scala, R, SQL |
| Workloads | Batch only | Batch + streaming + SQL + ML + graph |
| Iterative algorithms | Very slow (re-reads from disk each pass) | Excellent — data cached in memory |
# PySpark — the same operations you know from Pandas, distributed across a cluster
# from pyspark.sql import SparkSession
# from pyspark.sql import functions as F
#
# spark = SparkSession.builder.appName("SalesAnalysis").getOrCreate()
#
# df = spark.read.parquet("s3://bucket/sales/*.parquet") # billions of rows
#
# result = (df
# .filter(F.col("year") == 2026)
# .groupBy("region", "category")
# .agg(F.sum("revenue").alias("total_revenue"),
# F.count("*").alias("orders"),
# F.avg("revenue").alias("avg_order"))
# .orderBy(F.desc("total_revenue")))
#
# result.show(20)
# result.write.mode("overwrite").parquet("s3://bucket/output/")
Big Data Challenges
| Challenge | Description |
|---|
| Storage and cost | Petabyte-scale storage plus replication is expensive |
| Data quality (veracity) | Cleaning at scale is far harder than cleaning a CSV |
| Integration | Merging dozens of heterogeneous sources with conflicting schemas |
| Security and privacy | Large centralised stores are high-value breach targets; GDPR/DPDP compliance |
| Skills gap | Shortage of engineers who know distributed systems and analytics |
| Real-time demands | Sub-second latency requirements over huge volumes |
| Governance | Lineage, cataloguing, access control, retention policies |
| Tool sprawl | The ecosystem changes fast; integration burden is high |
Benefits
| Benefit | Example |
|---|
| Better, evidence-based decisions | Real-time inventory and pricing decisions |
| Deep customer understanding | 360° customer view across every touchpoint |
| Operational efficiency | Predictive maintenance eliminating unplanned downtime |
| New revenue streams | Data-driven products and services |
| Real-time fraud detection | Blocking a fraudulent transaction in milliseconds |
| Personalisation at scale | Individual recommendations for millions of users |
| Innovation | Training the ML/AI models that need enormous datasets |
The next two lessons detail the foundational big data platform: Hadoop and its MapReduce programming model.
