13.2 Shared Memory and Distributed Memory Programming

Parallel programming harnesses multiple processors to tackle complex computations. Shared memory allows easy data sharing, while distributed memory requires explicit communication. These approaches offer different trade-offs in terms of scalability and programming complexity.

Performance is key in parallel computing. Metrics like speedup and efficiency help evaluate program effectiveness. Optimizing communication, load balancing, and synchronization are crucial for achieving peak performance across multiple processors.

Shared Memory Programming

Shared memory parallel programming

Top images from around the web for Shared memory parallel programming

• 

  Reading 23: Queues and Message-Passing View original

  Is this image relevant?

• 

  OpenMP - HPC Wiki View original

  Is this image relevant?

• 

  Parallel Programming - HPC Wiki View original

  Is this image relevant?

• 

  Reading 23: Queues and Message-Passing View original

  Is this image relevant?

• 

  OpenMP - HPC Wiki View original

  Is this image relevant?

1 of 3

Top images from around the web for Shared memory parallel programming

• 

  Reading 23: Queues and Message-Passing View original

  Is this image relevant?

• 

  OpenMP - HPC Wiki View original

  Is this image relevant?

• 

  Parallel Programming - HPC Wiki View original

  Is this image relevant?

• 

  Reading 23: Queues and Message-Passing View original

  Is this image relevant?

• 

  OpenMP - HPC Wiki View original

  Is this image relevant?

1 of 3

• OpenMP enables easy parallelization through directives, employs fork-join model where main thread spawns parallel regions (parallel sections, loops)
• Pthreads provide fine-grained control over thread creation, synchronization, and management, suitable for complex parallel algorithms
• Shared memory architecture allows multiple processors to access common memory space, facilitates data sharing and communication
• Thread creation involves spawning new threads of execution, management includes scheduling and termination
• Data sharing categorizes variables as private (thread-specific) or shared (accessible by all threads)
• Work distribution techniques divide computational tasks among threads (loop parallelization, task parallelism)
• Synchronization mechanisms prevent data races and ensure thread coordination (mutexes, barriers, atomic operations)

Distributed Memory Programming

Distributed memory parallel programming

• MPI standardizes message-passing communication across different platforms and languages
• Distributed memory architecture assigns separate memory spaces to each processor, requires explicit communication
• Process creation establishes multiple executing instances of a program
• Point-to-point communication facilitates direct data exchange between two processes (send, receive operations)
• Collective communication involves multiple processes simultaneously (broadcast, scatter, gather, reduce)
• Data partitioning divides problem data across processes, crucial for load balancing and scalability
• Parallel algorithm design patterns structure distributed computations (master-worker, pipeline, divide-and-conquer)

Parallel Programming Concepts

Concepts in parallel programming

• Synchronization coordinates thread execution, prevents race conditions and deadlocks
• Communication methods include message passing and shared memory access, consider latency and bandwidth
• Load balancing distributes workload evenly among processors, improves efficiency (static, dynamic, work stealing)
• Parallel overhead encompasses additional time for inter-process communication and synchronization
• Granularity refers to task size in parallel decomposition (fine-grained: many small tasks, coarse-grained: fewer large tasks)

Performance of parallel programs

• Performance metrics quantify parallel program efficiency (speedup, efficiency, Amdahl's Law)
• Scalability analysis evaluates performance as problem size or processor count increases (strong scaling, weak scaling)
• Bottleneck identification pinpoints performance limitations using profiling tools and analysis techniques
• Optimization strategies enhance parallel performance (minimizing communication, improving load balance, reducing synchronization)
• Parallel efficiency measures resource utilization (processor, memory bandwidth)
• Performance modeling predicts and analyzes parallel program behavior (Roofline model, LogP model)