5.1 Memory hierarchy and locality of reference

Memory hierarchy is a crucial concept in computer architecture. It organizes memory into levels with different speeds and capacities, from fast registers to slower storage. This design exploits locality of reference, where programs tend to access a small portion of memory frequently.

By using faster, smaller memory for frequently accessed data, memory hierarchy bridges the gap between CPU speed and slower memory. This creates the illusion of a large, fast, and cheap memory system, enabling efficient performance in modern computers.

Memory Hierarchy and its Significance

Understanding Memory Hierarchy

Top images from around the web for Understanding Memory Hierarchy

• 

  Cluster computing — Introduction to Geodynamic Modelling 2018 documentation View original

  Is this image relevant?

• 

  Memory hierarchy - Wikipedia View original

  Is this image relevant?

• 

  HPC-Dictionary - HPC Wiki View original

  Is this image relevant?

• 

  Cluster computing — Introduction to Geodynamic Modelling 2018 documentation View original

  Is this image relevant?

• 

  Memory hierarchy - Wikipedia View original

  Is this image relevant?

1 of 3

Top images from around the web for Understanding Memory Hierarchy

• 

  Cluster computing — Introduction to Geodynamic Modelling 2018 documentation View original

  Is this image relevant?

• 

  Memory hierarchy - Wikipedia View original

  Is this image relevant?

• 

  HPC-Dictionary - HPC Wiki View original

  Is this image relevant?

• 

  Cluster computing — Introduction to Geodynamic Modelling 2018 documentation View original

  Is this image relevant?

• 

  Memory hierarchy - Wikipedia View original

  Is this image relevant?

1 of 3

• Memory hierarchy organizes memory into multiple levels (registers, cache, main memory, secondary storage) with different characteristics (capacity, access time, cost)
• Designed to exploit the principle of locality stating programs tend to access a small portion of their address space at any given time (temporal and spatial locality)
• Provides the illusion of a large, fast, and inexpensive memory system by combining multiple levels of memory with different characteristics
• Crucial for achieving high performance in computer systems by reducing the average memory access time and minimizing the overall memory system cost

Importance of Memory Hierarchy

• Enables efficient access to frequently used data and instructions by storing them in faster, smaller memory levels (registers, cache)
• Reduces the performance gap between the CPU and slower memory levels (main memory, secondary storage) by exploiting locality principles
• Allows for cost-effective memory systems by using a combination of expensive, fast memory (registers, cache) and cheaper, slower memory (main memory, secondary storage)
• Facilitates the development of complex, memory-intensive applications by providing a large, fast, and inexpensive memory system abstraction to programmers

Levels of Memory Hierarchy

Registers and Cache Memory

• Registers are the fastest and most expensive memory, located closest to the CPU
  • Typical access time of less than 1 nanosecond and a capacity of a few hundred bytes
  • Used for storing frequently accessed data and instructions during CPU operations
• Cache memory is a small, fast memory located between the CPU and main memory
  • Designed to store frequently accessed data and instructions
  • Access time of a few nanoseconds and a capacity of several kilobytes to a few megabytes
  • Organized into multiple levels (L1, L2, L3), each with increasing capacity and access time

Main Memory and Secondary Storage

• Main memory, also known as Random Access Memory (RAM), is the primary working memory of a computer system
  • Access time of tens of nanoseconds and a capacity of several gigabytes
  • Stores the currently executing programs, their data, and intermediate results
• Secondary storage, such as hard disk drives (HDDs) and solid-state drives (SSDs), has the largest capacity but the slowest access time
  • Access time typically in the range of milliseconds
  • Used for long-term storage of data and programs
  • Provides non-volatile storage, retaining data even when the system is powered off

Locality of Reference and Performance

Temporal and Spatial Locality

• Temporal locality refers to the tendency of a program to access the same memory locations repeatedly within a short period of time
  • Exploited by keeping recently accessed data in faster memory levels (cache)
  • Example: loops that access the same variables multiple times
• Spatial locality refers to the tendency of a program to access memory locations that are close to each other
  • Exploited by fetching and storing data in blocks, as nearby data is likely to be accessed in the near future
  • Example: accessing elements of an array sequentially

Optimizing Memory System Performance

• Effective use of locality principles in memory hierarchy design can significantly improve system performance
  • Reduces the number of accesses to slower memory levels
  • Minimizes the average memory access time
• Techniques such as prefetching and cache optimization can further enhance memory system performance
  • Prefetching anticipates future memory accesses and fetches data into faster memory levels before it is needed
  • Cache optimization techniques (block size, associativity, replacement policies) aim to maximize cache hit rates and minimize cache misses

Trade-offs in Memory Hierarchy

Cost, Capacity, and Access Time

• Moving from the top (registers) to the bottom (secondary storage) of the memory hierarchy:
  • Cost per bit decreases
  • Capacity increases
  • Access time becomes longer
• Trade-off between cost and performance is a key factor in determining the size and number of levels in the memory hierarchy
  • Faster memory technologies (SRAM) are more expensive, limiting their capacity in a cost-effective system
  • Slower memory technologies (DRAM, HDDs) are cheaper, allowing for larger capacities

Balancing Performance and Cost

• Access time gap between adjacent levels of the memory hierarchy is critical for overall system performance
  • A larger gap can result in significant performance penalties when accessing data from slower levels
  • Example: cache miss penalty, where the CPU has to wait for data to be fetched from main memory
• Balancing capacity and access time at each level is crucial for optimizing system performance
  • Too small a capacity at a given level can result in frequent accesses to slower levels
  • Too large a capacity can be cost-prohibitive and underutilized
• Advanced caching techniques, such as multi-level caches and cache hierarchies, can help bridge the access time gap between memory levels
  • Example: using a combination of small, fast L1 cache and larger, slower L2 cache to balance performance and cost