Siksha Sarovar

Siksha Sarovar (sikshasarovar.com) is a free educational web application that helps students in India learn programming and prepare for academic and competitive exams. The platform offers structured coding courses (C, C++, Python, Java, HTML, CSS, PHP, Power BI, AI, Machine Learning, Data Science), complete university curriculum notes for BCA/MCA students with previous year question papers, Class 10 and Class 12 CBSE/HBSE school notes, and dedicated preparation material for SSC, UPSC, Banking, Railway and other government exams. Browsing the site is completely free and requires no account. Users may optionally sign in with Google solely to save their learning progress, quiz scores and personal preferences across devices.

Privacy Policy | Terms of Service | Contact Siksha Sarovar | About Siksha Sarovar

v4.0.9 · PWA
Siksha Sarovar logo
Siksha Sarovar
Your Learning Universe

Siksha Sarovar is a free e-learning platform for coding courses, BCA university notes and competitive exam preparation. Optional Google sign-in saves your learning progress across devices.

Initializing knowledge base…
Compiling modules 0%

Parallel vs. Distributed Computing

Lesson 18 of 30 in the free Cloud Computing notes on Siksha Sarovar, written by Rohit Jangra.

Parallel vs. Distributed Computing

Though often used interchangeably, parallel computing and distributed computing are distinct paradigms. Modern cloud systems routinely combine both, so understanding the difference is essential.

Parallel Computing

Parallel computing uses multiple processors or cores within a single machine (or a tightly coupled system) to solve a problem faster by dividing work into sub-tasks that execute simultaneously.

  • Key resource: Shared memory bus, high-speed interconnect
  • Goal: Minimize time to solution for a fixed problem
  • Example: A 64-core server rendering a 3D scene — each core handles a tile of pixels

Distributed Computing

Distributed computing coordinates many independent machines (nodes) over a network to collectively solve a problem or provide a service.

  • Key resource: Network (LAN/WAN); nodes have private memory
  • Goal: Scale beyond one machine; tolerate failures; serve geographically dispersed users
  • Example: Apache Hadoop processing a 100 TB dataset across 500 commodity servers

Side-by-Side Comparison

PropertyParallel ComputingDistributed Computing
Memory modelShared (or tightly coupled)Private per node (message passing)
CommunicationVery fast (bus/cache coherence)Slower (TCP/IP, RPCs)
Fault toleranceLow (one machine = single point of failure)High (nodes can fail independently)
ScaleHundreds of cores (vertical)Thousands of nodes (horizontal)
LatencyNanoseconds to microsecondsMilliseconds (network round-trip)

Two Models, Visualized

When to Use Each

ScenarioPreferred Paradigm
Real-time video encodingParallel (GPU, shared memory)
Global e-commerce websiteDistributed (geo-replicated nodes)
Scientific simulation (CFD)Both (MPI across nodes, OpenMP within)
Machine learning trainingBoth (data parallelism across GPUs + distributed parameter server)

Cloud Systems Use Both

AWS HPC clusters run MPI jobs where each node uses multi-core parallel processing internally while communicating across nodes via Elastic Fabric Adapter (EFA). Google Spanner is a globally distributed database that uses Paxos consensus (distributed) while each shard uses parallel query execution (parallel). The two paradigms are complementary, not competing.

Summary

  • Parallel = speed through simultaneous local computation.
  • Distributed = scale and resilience through networked cooperation.
  • Cloud = both, applied at the right layer.