[Paper Review] METAQ: Bundle Supercomputing Tasks

Evan Berkowitz|arXiv (Cornell University)|Feb 20, 2017

Distributed and Parallel Computing Systems14 citations

TL;DR

METAQ is a lightweight bash-based system that bundles HPC tasks into larger jobs to exploit discounts and incentives for large allocations, enabling dynamic backfilling of shorter tasks to reduce idle time. It separates job submission from task description, allowing collaborative, efficient resource use and reducing wasted compute cycles by up to 75% compared to naive bundling.

ABSTRACT

We describe a light-weight system of bash scripts for efficiently bundling supercomputing tasks into large jobs, so that one can take advantage of incentives or discounts for requesting large allocations. The software can backfill computational tasks, avoiding wasted cycles, and can streamline collaboration between different users. It is simple to use, functioning similarly to batch systems like PBS, MOAB, and SLURM.

Motivation & Objective

• To address inefficiencies in supercomputing task scheduling caused by variable runtimes and underutilized nodes when bundling small tasks.
• To enable users to leverage large-job incentives and discounts without sacrificing performance or throughput.
• To streamline collaboration by decoupling task definition from job submission, allowing multiple users to contribute tasks to shared jobs.
• To reduce wasted computational cycles through dynamic backfilling of shorter tasks within larger job allocations.
• To provide a simple, portable, and freely available tool that integrates with existing HPC batch systems like SLURM, PBS, and MOAB.

Proposed method

• Users write task scripts with resource and environment specifications, stored in designated folders, separate from job scripts.
• A job script is submitted to the batch scheduler, specifying the requested allocation size and METAQ configuration.
• METAQ dynamically launches tasks from the task folder on the allocated nodes, managing execution and resource use.
• The system enables backfilling by interleaving shorter tasks within idle time of longer-running tasks, minimizing node idleness.
• It supports collaboration by allowing multiple users to submit job scripts that consume tasks from shared task folders.
• METAQ logs all task and system behavior, with manual cleanup available via date-based deletion of old logs and jobs.

Experimental results

Research questions

• RQ1 How can HPC workloads be efficiently bundled to reduce idle time and improve utilization of large allocations?
• RQ2 To what extent can dynamic backfilling of individual tasks within a job reduce wasted compute cycles compared to naive bundling?
• RQ3 How can collaboration across multiple users be streamlined in HPC environments with heterogeneous schedulers and resource policies?
• RQ4 What performance and scalability trade-offs arise when using lightweight bash scripts versus lower-level task managers?
• RQ5 Can a simple, portable system effectively manage complex, large-scale HPC workloads without requiring changes to existing HPC infrastructure?

Key findings

• Naïve bundling of similar 8-node tasks resulted in approximately 25% wasted compute time due to runtime variability and performance differences.
• Using METAQ for dynamic backfilling reduced idle time to around 5%, representing a 75% improvement in resource efficiency.
• The effective computational speedup achieved through reduced idle time is approximately 1.27×, equivalent to a 27% performance gain.
• METAQ enabled the CalLat collaboration to efficiently consume a 65 million node-hour INCITE allocation in about 5 months using jobs of 256–512 nodes.
• The system allowed multiple collaborators to submit job scripts that consumed shared task scripts, improving collaboration and priority accumulation without interrupting ongoing work.
• Despite initial service node overload on Titan due to high process count, the system proved robust and scalable when used within revised resource limits.