[Paper Review] Mistral Supercomputer Job History Analysis

Michał Zasadziński, Víctor Muntés-Mulero+2 more|arXiv (Cornell University)|Jan 23, 2018

AI-based Problem Solving and Planning1 references3 citations

TL;DR

This paper analyzes job history from the Mistral petascale supercomputer (3.14 PFLOPS, 3,300 nodes) using Slurm scheduler data, hardware metrics, and power monitoring to identify failure patterns. It reveals that failed jobs consume significantly more CPU hours than completed ones, use more nodes and racks (especially around 7 racks), and show lower average power consumption, suggesting early detection of failures could yield major resource savings in HPC environments.

ABSTRACT

In this technical report, we show insights and results of operational data analysis from petascale supercomputer Mistral, which is ranked as 42nd most powerful in the world as of January 2018. Data sources include hardware monitoring data, job scheduler history, topology, and hardware information. We explore job state sequences, spatial distribution, and electric power patterns.

Motivation & Objective

• Investigate failure patterns in large-scale HPC systems to improve resource utilization and system reliability.
• Identify correlations between job characteristics (e.g., number of nodes, racks, duration) and failure states in a petascale supercomputer.
• Analyze power consumption trends across different job completion states to detect anomalies indicating impending failure.
• Understand the impact of job size and hardware configuration on failure likelihood and duration.
• Support root cause analysis for complex IT systems by leveraging operational data from a real-world HPC environment.

Proposed method

• Collected and processed job scheduler logs (Slurm) from 10 months of production use on the Mistral supercomputer.
• Integrated hardware monitoring data, power metrics, and topology information (racks, chassis, node types) from the DKRZ data center.
• Analyzed job state sequences, spatial distribution (racks and nodes), and power consumption patterns across job types.
• Grouped jobs by hardware profile (e.g., B720_36_64, B720_24_256) and computed average power in the last 300 seconds before completion.
• Used statistical analysis and visualization (e.g., plots of failure probability vs. number of racks) to detect trends.
• Evaluated correlations between job priority, duration, and failure states, identifying anomalies such as high-priority timeouts.

Experimental results

Research questions

• RQ1 What is the relationship between the number of allocated nodes, racks, and the likelihood of job failure?
• RQ2 How do power consumption patterns differ between completed, failed, and cancelled jobs, especially in the final 300 seconds?
• RQ3 Does job duration or size correlate with failure likelihood, and are there specific thresholds where failure probability increases?
• RQ4 Are there detectable anomalies in job priority distribution for specific failure states, such as timeouts?
• RQ5 How does spatial distribution (e.g., number of racks used) influence failure occurrence and duration?

Key findings

• Completed jobs accounted for 91.3% of all submissions, while failed jobs made up 5.6%, with 1.7% cancelled and 1.4% timing out.
• Failed steps used an average of 18 nodes, compared to 3.4 for completed steps, indicating higher complexity in failing jobs.
• The probability of failure peaks at 7 racks used, with failures becoming rare for allocations over 13 racks (over 1,000 nodes).
• For steps using fewer than 10 racks, failed jobs were shorter in duration than completed ones; for more than 12 racks, failed jobs were significantly longer.
• Failed jobs consumed more CPU hours than completed ones, with average durations of ~414s for completed steps and nearly triple that for failed steps.
• Power consumption in the final 300 seconds was consistently lower for failed jobs (e.g., 172W for B720-compute_36_64) than for completed jobs (176W), suggesting early idle states after failure.