Approaches to Parallel Programming

Parallel hardware is useless without programming models that let developers express parallelism. Four major paradigms cover the landscape, each suited to different hardware and problem structures.

Taxonomy of Approaches

1. Shared Memory Programming

All threads access a common address space. Synchronization (mutexes, semaphores, barriers) prevents race conditions.

• OpenMP: Compiler directives for C/C++/Fortran. Add #pragma omp parallel for before a loop and the compiler parallelizes it across cores. Used in scientific computing.
• pthreads: Low-level POSIX thread API. Fine-grained control, high complexity.
• Java synchronized: Language-level shared memory concurrency.

Pros: Simple data sharing, low latency communication Cons: Limited to one machine; race conditions and deadlocks are hard to debug

2. Message Passing

Processes have private memory and communicate by explicitly sending and receiving messages. No shared state.

• MPI (Message Passing Interface): The standard for HPC. Programs run as N independent processes; data is exchanged via MPI_Send / MPI_Recv. Used on supercomputers and AWS HPC clusters running thousands of nodes.
• Actor model (Erlang, Akka): Each actor has a mailbox; actors communicate asynchronously.

Pros: Scales to thousands of nodes; no race conditions (no shared state) Cons: Programmer must manage all data movement explicitly

3. Data Parallel

Apply the same operation to many data elements simultaneously. Maps directly to SIMD hardware (GPUs, vector units).

• CUDA: NVIDIA's programming model for GPUs. A kernel function runs on thousands of GPU threads simultaneously, each processing a different data element.
• OpenCL: Vendor-neutral alternative (works on AMD, Intel, NVIDIA).
• CPU intrinsics (AVX-512): Explicit SIMD instructions in C/C++.

Use case: Deep learning (matrix multiply), image processing, physics simulations.

4. Task Parallel

Decompose the program into independent tasks that can run in any order. A runtime scheduler assigns tasks to available threads/cores.

• Intel TBB (Threading Building Blocks): C++ library with parallel_for, pipeline, task_group.
• Go goroutines: Lightweight green threads with channel-based communication.
• AWS Lambda: Serverless task parallelism at cloud scale — each invocation is an independent task.

Comparison

Cloud Mapping