13.1 Parallel Computing Architectures and Models

Parallel computing architectures revolutionize scientific computing by dividing tasks among multiple processors. From shared memory systems to GPU-based setups, each architecture offers unique advantages for tackling complex problems efficiently.

Performance in parallel computing hinges on scalability, measured by laws like Amdahl's and Gustafson's. Different architectures excel at specific scientific problems, from Monte Carlo simulations to climate modeling, enhancing computational power across various fields.

Parallel Computing Architectures

Types of parallel computing architectures

Top images from around the web for Types of parallel computing architectures

• 

  Microprocessor Design/GPU - Wikibooks, open books for an open world View original

  Is this image relevant?

• 

  Distributed computing - Wikipedia View original

  Is this image relevant?

• 

  Parallel Programming - HPC Wiki View original

  Is this image relevant?

• 

  Microprocessor Design/GPU - Wikibooks, open books for an open world View original

  Is this image relevant?

• 

  Distributed computing - Wikipedia View original

  Is this image relevant?

1 of 3

Top images from around the web for Types of parallel computing architectures

• 

  Microprocessor Design/GPU - Wikibooks, open books for an open world View original

  Is this image relevant?

• 

  Distributed computing - Wikipedia View original

  Is this image relevant?

• 

  Parallel Programming - HPC Wiki View original

  Is this image relevant?

• 

  Microprocessor Design/GPU - Wikibooks, open books for an open world View original

  Is this image relevant?

• 

  Distributed computing - Wikipedia View original

  Is this image relevant?

1 of 3

• Shared Memory Systems enable multiple processors to access common memory space facilitating fast inter-processor communication but limited scalability due to memory contention (OpenMP)
• Distributed Memory Systems allocate local memory to each processor communicating through message passing achieving high scalability with higher communication overhead (MPI)
• GPU-based Systems specialize in highly parallel computations utilizing thousands of simple cores for massive parallelism efficient for data-parallel tasks (CUDA, OpenCL)

Models of parallel computing

• Task Parallelism executes different tasks concurrently on multiple processors suitable for problems with independent subtasks (job scheduling, workflow management)
• Data Parallelism applies same operation to different data elements simultaneously efficient for large datasets with uniform operations (matrix multiplication, image processing)
• Hybrid Parallelism combines task and data parallelism exploiting benefits of both models for complex problems (climate modeling, computational fluid dynamics)

Performance and Application

Scalability in parallel computing

• Amdahl's Law calculates theoretical speedup limited by sequential portion of code $S(n) = \frac{1}{(1-p) + \frac{p}{n}}$ where $p$ is parallelizable fraction and $n$ is number of processors
• Gustafson's Law accounts for increased problem size with more processors $S(n) = n - \alpha(n - 1)$ where $\alpha$ is sequential fraction and $n$ is number of processors
• Strong Scaling measures performance with fixed problem size and increasing number of processors efficiency tends to decrease as processors increase
• Weak Scaling evaluates performance as problem size increases with number of processors aims to maintain constant efficiency

Architectures for scientific computing problems

• Shared Memory Systems suit problems with frequent inter-thread communication (Monte Carlo simulations, matrix operations)
• Distributed Memory Systems excel for problems with large datasets and minimal communication (climate modeling, particle simulations)
• GPU-based Systems efficiently handle problems with high arithmetic intensity (image processing, deep learning)
• Task Parallelism approaches problems with diverse independent subtasks (multi-physics simulations, parameter sweeps)
• Data Parallelism tackles problems with uniform operations on large datasets (numerical integration, Fourier transforms)