[Paper Review] A Workload Analysis of NSF's Innovative HPC Resources Using XDMoD

Nikolay A. Simakov, Joseph P. White+6 more|arXiv (Cornell University)|Jan 12, 2018

Distributed and Parallel Computing Systems2 references17 citations

TL;DR

This paper analyzes the workload of NSF's Innovative HPC resources from 2011 to 2017 using XDMoD, revealing high utilization (90% of allocations used), declining average job size from ~9,000 to ~1,000 cores, increasing single-node job usage (21% of SU consumption in 2017), and growing Science Gateway adoption—especially in biological and physical sciences—while identifying trends in memory use, parallelism, and I/O patterns across systems like TACC and SDSC clusters.

ABSTRACT

Workload characterization is an integral part of performance analysis of high performance computing (HPC) systems. An understanding of workload properties sheds light on resource utilization and can be used to inform performance optimization both at the software and system configuration levels. It can provide information on how computational science usage modalities are changing that could potentially aid holistic capacity planning for the wider HPC ecosystem. Here, we report on the results of a detailed workload analysis of the portfolio of supercomputers comprising the NSF Innovative HPC program in order to characterize its past and current workload and look for trends to understand the nature of how the broad portfolio of computational science research is being supported and how it is changing over time. The workload analysis also sought to illustrate a wide variety of usage patterns and performance requirements for jobs running on these systems. File system performance, memory utilization and the types of parallelism employed by users (MPI, threads, etc) were also studied for all systems for which job level performance data was available.

Motivation & Objective

• To characterize the evolving workload patterns across NSF’s Innovative HPC resources to inform system optimization and capacity planning.
• To understand shifts in computational science usage modalities, including job size, parallelism, and resource utilization trends.
• To assess the role of Science Gateways in HPC usage and their growing impact on user access and job throughput.
• To analyze system-level metrics such as memory, I/O, and interconnect usage to identify performance bottlenecks and optimization opportunities.
• To quantify the impact of architectural changes (e.g., TACC STAMPEDE2) on job characteristics and resource consumption patterns.

Proposed method

• Employed XDMoD (XSEDE Monitoring System) to collect and analyze job accounting and performance data from 2011–2017 across 8 major HPC systems.
• Used job-level metrics from /proc/stat, /sys/devices/system/node/, and /proc/sys/lnet/stats to derive CPU, memory, I/O, and interconnect usage.
• Applied Lariat and XALT tools to extract executable paths and classify applications based on known community software databases.
• Implemented a classification algorithm to identify the main job step by counting unique PIDs and filtering out system daemons.
• Used regular expressions and a reference database of 188 known applications to categorize jobs; applied machine learning to improve classification of uncategorized jobs.
• Computed derived metrics such as average and maximum memory usage, I/O rates (reads/writes), and bandwidth utilization from sub-sampled instantaneous data.

Experimental results

Research questions

• RQ1 How has the utilization of NSF’s HPC resources evolved from 2011 to 2017, and what factors drive changes in allocation usage?
• RQ2 What trends exist in job size, parallelism type (MPI, threads), and memory consumption across different scientific domains?
• RQ3 How has the adoption of Science Gateways influenced job throughput, user demographics, and resource consumption patterns?
• RQ4 To what extent are HPC applications bandwidth-limited, and how does interconnect usage correlate with job size and performance?
• RQ5 What is the impact of architectural changes (e.g., TACC STAMPEDE2) on job characteristics and system efficiency?

Key findings

• Overall allocation utilization was high, with only 10% of allocations going unused, and total usage across NSF directorates increased by two orders of magnitude since 2011.
• The Mathematical and Physical Sciences (MPS) and Biological Sciences (BIO) directorates accounted for 70% of all XD SUs consumed, with Molecular Biosciences, Physics, and Materials Research contributing half of that total.
• Single-node jobs increased from 5% of XD SUs in 2011 to 21% in 2017, and serial jobs accounted for at least 3% of all XD SUs consumed in 2017.
• Despite decreasing average job size, a recent trend with TACC STAMPEDE2 shows increasing use of large jobs (8,000+ cores), indicating a shift toward high-core workloads.
• Most parallel jobs efficiently use all or nearly all allocated CPU cores, and the majority use multiple single-threaded processes rather than multi-threaded ones.
• Science Gateway usage grew fivefold from 2011 to 2017, with 98% of Gateway SUs consumed by Biological and Mathematical/Physical Sciences, and Gateway users surpassed HPC users in number by 2015.