4.3.1 Handling Failures: The Resilience of Hadoop

Hadoop is built on the principle that "Failure is the norm."

1. Task Failure

• If a Mapper or Reducer crashes, the NM reports the failure to the AM.
• The AM will Reschedule the task on a different node.
• It will try up to 4 times (default) before failing the entire job.

2. NodeManager Failure

• If a NodeManager stops sending heartbeats to the RM, the RM marks the node as "down."
• All tasks that were running on that node are marked as failed and rescheduled.

3. ResourceManager Failure

• In modern Hadoop, we use High Availability (HA) with two ResourceManagers (Active and Standby). If Active fails, Standby takes over instantly using Zookeeper.

4.3.2 Job Scheduling in Hadoop

Since Big Data clusters are expensive, they are usually shared. We need a way to decide "Who goes first?"

4.3.3 Speculative Execution: The Straggler Solution

In a large cluster, some nodes might be "stragglers"—nodes that are much slower than others due to failing hardware or network congestion.

• Hadoop's Solution: Launch a duplicate "speculative" task on a different node.
• The Race: Whoever finishes first wins and the other task is killed.
• Benefit: This prevents one bad node from slowing down a job that involves 1,000 other healthy nodes.
• JVM Reuse: Instead of starting a new Java Virtual Machine for every tiny task, Hadoop can reuse the same JVM, saving startup overhead.