[Paper Review] Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchip

Luigi Fusco, Mikhail Khalilov+4 more|arXiv (Cornell University)|Aug 21, 2024

Modular Robots and Swarm Intelligence4 citations

TL;DR

This paper presents a comprehensive performance characterization of the Quad GH200 node in the Alps supercomputer, analyzing memory access patterns across CPUs, GPUs, and heterogeneous memory types (HBM, DDR). Using custom microbenchmarks, it demonstrates that data placement and locality—especially within the same Grace Hopper Superchip—significantly impact performance, with C2C interconnect enabling high-bandwidth, low-latency access that unlocks performance for memory-bound workloads like LLM inference and NCCL collectives.

ABSTRACT

Heterogeneous supercomputers have become the standard in HPC. GPUs in particular have dominated the accelerator landscape, offering unprecedented performance in parallel workloads and unlocking new possibilities in fields like AI and climate modeling. With many workloads becoming memory-bound, improving the communication latency and bandwidth within the system has become a main driver in the development of new architectures. The Grace Hopper Superchip (GH200) is a significant step in the direction of tightly coupled heterogeneous systems, in which all CPUs and GPUs share a unified address space and support transparent fine grained access to all main memory on the system. We characterize both intra- and inter-node memory operations on the Quad GH200 nodes of the new Swiss National Supercomputing Centre Alps supercomputer, and show the importance of careful memory placement on example workloads, highlighting tradeoffs and opportunities.

Motivation & Objective

• To understand the performance characteristics of data movement in tightly coupled heterogeneous systems, focusing on the Grace Hopper Superchip (GH200).
• To evaluate the impact of memory allocation policies and data placement strategies on real-world workloads such as LLM inference and NCCL collectives.
• To quantify the performance differences between intra- and inter-node memory operations across various memory types (HBM, DDR) and processing units (CPU, GPU).
• To validate that the NVLink-C2C interconnect enables effective, low-latency, cache-coherent access across heterogeneous PUs, enabling unified memory access.

Proposed method

• Designing and executing a comprehensive suite of microbenchmarks to measure read, write, and copy operations across all combinations of processing units (CPU, GPU) and memory types (HBM, DDR).
• Using a datapath-oriented approach to correlate measured performance with theoretical bandwidth and latency limits of the underlying hardware interconnects (NVLink-C2C, Slingshot).
• Employing PyTorch’s pluggable allocator to control memory allocation on HBM (GPU memory) and NUMA-aware allocations on DDR (CPU memory), enabling controlled experiments on data placement.
• Measuring performance of NCCL all-reduce and all-gather operations across intra-node and inter-node configurations to assess scalability and locality effects.
• Analyzing LLM inference workloads (Llama2-7b and Llama2-13b) with varying memory allocation strategies to evaluate real-world performance implications.
• Validating results by comparing observed performance against theoretical bounds derived from the system’s physical datapaths and interconnect capabilities.

Figure 2: Maximum bandwidth plotted against access latency achieved by Grace (left) and Hopper (right) to different memories of the system. The suffix ”-p” indicates memory on a peer GH200.

Experimental results

Research questions

• RQ1 How does data placement within the GH200 node—across CPU, GPU, HBM, and DDR—impact memory access performance for synthetic and application workloads?
• RQ2 To what extent does the NVLink-C2C interconnect enable low-latency, high-bandwidth, cache-coherent access between CPUs and GPUs in a tightly coupled system?
• RQ3 How do memory access patterns and allocation policies affect the performance of memory-bound workloads such as LLM inference and NCCL collectives?
• RQ4 What is the relative performance of intra-Superchip versus inter-Superchip memory access, and how does it affect collective communication operations?

Key findings

• Intra-Superchip memory access (within the same GH200) provides significantly higher bandwidth and lower latency than peer access across different Superchips, especially for HBM and DDR.
• For LLM inference, memory access speed has a measurable impact on performance, with optimal data placement reducing inference time by up to 20% compared to suboptimal configurations.
• NCCL all-reduce and all-gather operations show that peer DDR access severely limits scalability, while same-GH200 access achieves near-peak performance regardless of memory type (HBM or DDR).
• The C2C interconnect enables transparent, high-bandwidth access to all system memory from any PU, with performance approaching theoretical limits when data is placed close to the processing unit.
• Workloads using Tensor Cores become memory-bound when matrices are moved from HBM to DDR, with performance dropping significantly even with high-bandwidth memory.
• For large allocations, the PyTorch pluggable allocator incurs synchronization overhead, slowing performance compared to direct cudaMalloc, but enables controlled memory placement experiments.

Figure 3: Theoretical bandwidth bound for (left to right) read and write operations, copy operations issued by a Grace, and copy operations issued by a Hopper. Bounds depend on the datapath, which in turn depends on the types of memories involved. The bounds are shown in GB/s and include the limitin