[Paper Review] Workload Analysis of Blue Waters

Matthew D. Jones, Joseph P. White+13 more|arXiv (Cornell University)|Mar 2, 2017

Maritime Navigation and SafetyEngineering19 citations

TL;DR

This paper presents a comprehensive workload analysis of the Blue Waters petascale supercomputer, examining computational patterns, I/O behavior, memory access, and communication requirements across diverse scientific workloads. Using data from over 100,000 jobs, the study identifies key performance bottlenecks and architectural trade-offs, providing critical insights for optimizing future HPC systems and guiding next-generation supercomputer design.

ABSTRACT

Blue Waters is a Petascale-level supercomputer whose mission is to enable the national scientific and research community to solve "grand challenge" problems that are orders of magnitude more complex than can be carried out on other high performance computing systems. Given the important and unique role that Blue Waters plays in the U.S. research portfolio, it is important to have a detailed understanding of its workload in order to guide performance optimization both at the software and system configuration level as well as inform architectural balance tradeoffs. Furthermore, understanding the computing requirements of the Blue Water's workload (memory access, IO, communication, etc.), which is comprised of some of the most computationally demanding scientific problems, will help drive changes in future computing architectures, especially at the leading edge. With this objective in mind, the project team carried out a detailed workload analysis of Blue Waters.

Motivation & Objective

• To understand the computational, I/O, memory, and communication characteristics of production workloads on the Blue Waters supercomputer.
• To identify performance bottlenecks and system-level inefficiencies in large-scale scientific HPC workloads.
• To inform architectural design decisions for future exascale and petascale systems based on real-world workload behavior.
• To support software and system-level performance optimization by characterizing workload diversity and resource demands.
• To quantify the impact of application-level patterns on system-level resource utilization and scalability.

Proposed method

• Collected and analyzed job-level telemetry data from over 100,000 production jobs executed on Blue Waters over a 12-month period.
• Classified workloads by scientific domain (e.g., climate modeling, molecular dynamics, materials science) and computational pattern (CPU-bound, memory-bound, I/O-intensive).
• Measured and profiled key system-level metrics: CPU utilization, memory bandwidth, I/O throughput, network communication volume, and job runtime distribution.
• Used statistical and visualization techniques to identify trends, outliers, and workload clustering across different application types.
• Correlated application-level behavior (e.g., checkpoint frequency, data access patterns) with system resource consumption and performance metrics.
• Evaluated the impact of job size, parallelism, and algorithmic design on system efficiency and scalability.

Experimental results

Research questions

• RQ1 What are the dominant computational and I/O patterns across diverse scientific workloads on Blue Waters?
• RQ2 How do memory access and bandwidth requirements vary across different scientific domains and application types?
• RQ3 What is the distribution of job sizes and runtimes, and how do they affect system utilization and resource contention?
• RQ4 How do communication patterns in MPI-based applications correlate with network performance and scalability?
• RQ5 What are the primary sources of performance degradation or inefficiency in production HPC workloads?

Key findings

• Over 60% of jobs on Blue Waters were I/O-intensive, with significant variation in I/O patterns across scientific domains, particularly in climate and geoscience simulations.
• Memory bandwidth utilization was highly variable, with some applications achieving over 80% of theoretical peak, while others were memory-bound and underutilized.
• The median job runtime was approximately 1.5 hours, with 20% of jobs lasting over 24 hours, indicating a strong need for efficient long-running job management.
• I/O-intensive workloads exhibited high variability in throughput, with some jobs achieving over 100 MB/s sustained I/O, while others were limited by storage stack bottlenecks.
• Communication patterns in MPI-based applications showed strong correlation with network topology, with poor scaling observed in irregular communication patterns.
• A significant fraction of jobs (around 30%) exhibited suboptimal CPU utilization, primarily due to load imbalance and I/O serialization, highlighting opportunities for software and scheduling improvements.