[Paper Review] Enabling Machine Learning-Ready HPC Ensembles with Merlin

Peterson Jl, Bay B+18 more|arXiv (Cornell University)|Dec 5, 2019

Scientific Computing and Data Management21 references4 citations

TL;DR

Merlin is a high-performance, workflow-based framework that enables large-scale machine learning (ML)-ready ensembles of scientific HPC simulations by orchestrating distributed, heterogeneous workloads with low overhead. It supports dynamic, ML-augmented workflows—such as active learning and surrogate modeling—across leadership-class HPC systems, achieving up to 1 million ICF simulations per hour and enabling scalable, fault-tolerant analysis of massive datasets like the 100M JAG simulation ensemble.

ABSTRACT

With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. In this paper, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute technologies, Merlin aims to lower the barrier for scientific subject matter experts to incorporate ML into their analysis. In addition to its design, we describe some example applications that Merlin has enabled on leadership-class HPC resources, such as the ML-augmented optimization of nuclear fusion experiments and the calibration of infectious disease models to study the progression of and possible mitigation strategies for COVID-19.

Motivation & Objective

• To address the growing need for large-scale, ML-friendly simulation ensembles in scientific HPC due to the increasing complexity and volume of simulation and experimental data.
• To overcome the inherent tension between ML’s demand for millions of training samples and HPC’s optimization for few, long-running jobs.
• To lower the barrier for scientific domain experts to integrate ML into HPC workflows through a flexible, shell-like interface and automated task management.
• To enable scalable, fault-tolerant execution of heterogeneous, multi-stage workflows involving simulators, post-processors, and ML trainers across distributed HPC resources.
• To support real-time, dynamic sampling and optimization workflows using ML feedback to guide simulation selection, reducing computational waste.

Proposed method

• Merlin uses a dynamic hierarchical task-generation algorithm to scale simulation ensembles to several million samples with minimal overhead.
• It integrates with HPC job schedulers and uses a producer-consumer model to coordinate asynchronous workers across thousands of nodes.
• The framework leverages a shell-like command-line interface (CLI) for ease of use, enabling users to compose complex workflows with familiar scripting patterns.
• Merlin employs a provenance and resubmission system to recover from random failures without restarting entire workflows.
• It supports integration with external tools like Flux and Conduit for job management and workflow coordination across multiple HPC systems.
• The system is built on open-source components and exposes a YAML-based workflow description (Maestro) and a companion spellbook for reusable workflow components.

Experimental results

Research questions

• RQ1 How can a workflow system efficiently manage the creation of massive, multi-component simulation ensembles suitable for training accurate ML models in HPC environments?
• RQ2 What architectural patterns enable low-overhead, scalable execution of heterogeneous, asynchronous workflows involving simulators, post-processors, and ML trainers?
• RQ3 How can ML be integrated into HPC workloads to dynamically guide simulation selection and reduce computational waste through active learning?
• RQ4 To what extent can a workflow framework like Merlin scale to millions of simulation samples while maintaining low task queuing and execution overhead?
• RQ5 How can fault tolerance and provenance tracking be implemented in large-scale HPC workflows to enable recovery from node failures without restarting from scratch?

Key findings

• Merlin successfully scaled to create a 100M JAG dataset of 100 million ICF simulations, processing approximately 1 million simulations per hour on the Sierra HPC system.
• The framework achieved low overhead in task generation and execution, enabling efficient scaling to several million samples in idealized benchmark workloads.
• Merlin’s dynamic, ML-augmented workflow for ICF optimization reduced the number of required simulations by using active learning to guide new simulation selection.
• The epicast COVID-19 study used Merlin to coordinate multiple HPC systems into a unified resource, enabling high-throughput, time-sensitive analysis of infectious disease models.
• Merlin’s provenance and resubmission system allowed users to recover from random system failures in real time or post-facto, without requiring new dedicated compute allocations.
• The framework has been deployed at multiple HPC centers and is publicly available under the MIT license, with the 10,000-sample JAG dataset and a spellbook of reusable components also released.