Apache Hadoop

Apache Hadoop is an open-source framework for the distributed storage and distributed processing of very large datasets across clusters of commodity hardware. Created by Doug Cutting and Mike Cafarella in 2006 (named after Cutting's son's toy elephant), it was inspired by two Google papers — the Google File System (2003) and MapReduce (2004).

Core Design Principles

1. Scale out, not up — add cheap commodity machines rather than buying bigger servers
2. Assume failure is normal — with thousands of machines, something is always broken; handle it in software
3. Move computation to the data — sending a small program to where data lives beats moving terabytes across the network (data locality)
4. Write once, read many — optimised for large sequential reads, not random updates
5. Hide distribution complexity — the programmer writes logic, the framework handles parallelism, scheduling, and recovery

The Four Core Modules

1. HDFS — Hadoop Distributed File System

HDFS stores very large files by splitting them into blocks distributed across the cluster, with each block replicated for fault tolerance.

HDFS Architecture Components

Blocks and Replication

Default block size:  128 MB  (Hadoop 2+;  64 MB in Hadoop 1)
Default replication: 3

Example: a 500 MB file
   Block 1: 128 MB
   Block 2: 128 MB
   Block 3: 128 MB
   Block 4: 116 MB          <- the last block uses only what it needs

   Each of the 4 blocks is stored on 3 different DataNodes
   ->  12 block replicas distributed across the cluster
   ->  actual disk consumed = 500 MB x 3 = 1.5 GB

Why such large blocks? To minimise the seek-time-to-transfer-time ratio. With 128 MB blocks, disk seek overhead becomes negligible relative to the sequential read, and the NameNode's metadata stays small.

Rack Awareness — The Default Replica Placement Policy

   Replica 1: on the same node as the writer (or a random node if the client
              is outside the cluster)
   Replica 2: on a DIFFERENT RACK
   Replica 3: on a different node in the SAME rack as replica 2

Why this policy:
   • Surviving a whole-rack failure (replicas exist on ≥2 racks)
   • Minimising cross-rack network traffic (only 1 cross-rack write)
   • Balanced read load

HDFS Read and Write Flows

WRITE:
  1. Client asks the NameNode to create the file
  2. NameNode checks permissions and returns a list of DataNodes for block 1
  3. Client writes to DataNode A, which PIPELINES to B, which pipelines to C
  4. Acknowledgements flow back along the pipeline
  5. Repeat for each block; client tells the NameNode the file is complete

READ:
  1. Client asks the NameNode for the block locations of a file
  2. NameNode returns DataNode addresses, sorted by network proximity
  3. Client reads each block DIRECTLY from the nearest DataNode
     (data never flows through the NameNode — it would be a bottleneck)

HDFS Characteristics

Common HDFS Commands

hdfs dfs -ls /user/data                       # list a directory
hdfs dfs -mkdir /user/data/sales              # create a directory
hdfs dfs -put local_file.csv /user/data/      # upload from local FS
hdfs dfs -get /user/data/output.csv ./        # download to local FS
hdfs dfs -cat /user/data/file.txt             # print file contents
hdfs dfs -tail /user/data/file.txt            # last KB of a file
hdfs dfs -rm -r /user/data/old                # delete recursively
hdfs dfs -du -h /user/data                    # disk usage, human readable
hdfs dfs -setrep -w 2 /user/data/file.csv     # change replication factor
hdfs dfsadmin -report                         # cluster health report
hdfs fsck / -files -blocks                    # filesystem check

2. YARN — Yet Another Resource Negotiator

Introduced in Hadoop 2, YARN separated resource management from the processing model, turning Hadoop from a MapReduce-only system into a general-purpose cluster operating system.

Schedulers: FIFO (simple, unfair), Capacity Scheduler (guaranteed queue capacities per organisation), and Fair Scheduler (all jobs get an equal share over time).

3. MapReduce

The original processing engine — covered in full in the next lesson.

The Hadoop Ecosystem

Ecosystem Component Reference

Hive vs Pig vs HBase

Hadoop 1 vs Hadoop 2 vs Hadoop 3

Advantages and Limitations of Hadoop

The next lesson explains the programming model at Hadoop's heart: MapReduce.