Lazy home migration for distributed shared memory systems

Abstract

In a distributed shared memory system, each memory page is associated with a home node that maintains the directory state for cache lines within that page. Memory access patterns and home node locations have a strong influence on performance, especially if remote communication is costly. Since access patterns are difficult to predict and may change dynamically, it is useful to dynamically migrate home nodes to reduce the amount of remote communication. This paper presents a new and efficient algorithm for migrating home nodes in distributed shared memory systems. Unlike previous page migration algorithms, our algorithm avoids global coordination, allowing the system to be more responsive to changing workloads. We verify the algorithm's correctness with the Murφ protocol verification tool. We explore several policies for deciding when and where to migrate home nodes. Trace-driven simulations of several SPLASH-2 benchmarks show that our home migration algorithm and policies can reduce the amount of remote communication by 50%. The results also emphasize the importance of minimizing the cost of migration.