[Paper Review] Distributed, combined CPU and GPU profiling within HPX using APEX

Patrick Diehl, Gregor Daiß+7 more|arXiv (Cornell University)|Sep 21, 2022

Parallel Computing and Optimization Techniques4 citations

TL;DR

This paper presents a scalable, distributed CPU-GPU profiling framework using APEX within the HPX runtime to analyze performance bottlenecks in the Octo-Tiger astrophysics simulation across two supercomputing architectures—Piz Daint and Summit. It demonstrates that APEX enables low-overhead, large-scale profiling with measurable performance insights, revealing architectural differences and identifying optimization opportunities, particularly in GPU kernel execution and inter-node communication.

ABSTRACT

Benchmarking and comparing performance of a scientific simulation across hardware platforms is a complex task. When the simulation in question is constructed with an asynchronous, many-task (AMT) runtime offloading work to GPUs, the task becomes even more complex. In this paper, we discuss the use of a uniquely suited performance measurement library, APEX, to capture the performance behavior of a simulation built on HPX, a highly scalable, distributed AMT runtime. We examine the performance of the astrophysics simulation carried-out by Octo-Tiger on two different supercomputing architectures. We analyze the results of scaling and measurement overheads. In addition, we look in-depth at two similarly configured executions on the two systems to study how architectural differences affect performance and identify opportunities for optimization. As one such opportunity, we optimize the communication for the hydro solver and investigated its performance impact.

Motivation & Objective

• To enable large-scale, distributed profiling of CPU and GPU workloads in HPX-based scientific applications.
• To evaluate the feasibility and overhead of combined CPU-GPU profiling using APEX in real production-scale simulations.
• To identify performance bottlenecks in Octo-Tiger across different HPC architectures (Piz Daint and Summit).
• To optimize communication in the hydro solver based on profiling insights and measure its performance impact.
• To assess the scalability and accuracy of APEX profiling across up to 2000 nodes with GPU measurements.

Proposed method

• Integration of APEX with HPX to capture performance metrics across distributed nodes, including CPU and GPU workloads.
• Use of CUPTI for GPU profiling and HPX performance counters for CPU metrics, with data aggregation across compute nodes.
• Execution of a full production-scenario run of Octo-Tiger for 40 time-steps on 48 nodes (Piz Daint) and 8 nodes (Summit), each with 48 GPUs.
• Comparison of profiling overhead in both CPU-only and CPU-GPU runs to isolate GPU measurement impact.
• Analysis of runtime behavior, including kernel execution times, communication overhead, and task scheduling across architectures.
• Optimization of the hydro solver’s communication pattern based on APEX-identified bottlenecks, followed by re-profiling to measure improvement.

Experimental results

Research questions

• RQ1 How does the profiling overhead of APEX behave in large-scale, distributed CPU-GPU runs on modern HPC systems?
• RQ2 What are the key performance differences in Octo-Tiger’s execution between Piz Daint (Intel Xeon + P100) and Summit (Power9 + V100) architectures?
• RQ3 Which components of Octo-Tiger exhibit the highest performance overhead, and can they be optimized based on profiling data?
• RQ4 How does the transition from P100 to V100 GPUs affect average kernel execution times, and what factors contribute to the observed speedups?
• RQ5 To what extent does GPU profiling with CUPTI significantly increase overhead, especially at scale?

Key findings

• APEX enables scalable, distributed CPU-GPU profiling in HPX-based simulations, with feasible overhead even at 2000 nodes on Piz Daint.
• The profiling overhead on Piz Daint reached 52.4% at scale, primarily due to GPU measurements via CUPTI, while CPU-only runs showed lower overhead.
• On Summit, the profiling overhead was 31.8% at scale, indicating better scalability and lower impact for GPU profiling.
• The average GPU kernel runtime on V100 showed significant speedups over P100, likely due to increased L1 cache and memory bandwidth rather than SM count.
• The hydro solver’s communication was identified as a major bottleneck on Piz Daint, particularly in the schedule_parcel function, which was optimized based on APEX data.
• Post-optimization, APEX measurements confirmed performance improvements, demonstrating the value of distributed profiling for guiding code optimization in production HPC workloads.