[Paper Review] Africanus IV. The Stimela2 framework: scalable and reproducible workflows, from local to cloud compute

O. Smirnov, Sphesihle Makhathini+5 more|arXiv (Cornell University)|Dec 13, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

Stimela2 is a scalable, reproducible workflow framework for radio astronomy data reduction, using human-readable YAML recipes and containerized execution to enable portable, version-controlled processing from local machines to cloud environments like AWS. It supports modular, composable workflows with native integration to Slurm and Kubernetes, ensuring reproducibility through containerized, versioned dependencies and a certification model for certified workflows.

ABSTRACT

Stimela2 is a new-generation framework for developing data reduction workflows. It is designed for radio astronomy data but can be adapted for other data processing applications. Stimela2 aims at the middle ground between ease of development, human readability, and enabling robust, scalable and reproducible workflows. It represents workflows by linear, concise and intuitive YAML-format "recipes". Atomic data reduction tasks (binary executables, Python functions and code, and CASA tasks) are described by YAML-format "cab definitions" detailing each task's "schema" (inputs and outputs). Stimela2 provides a rich syntax for chaining tasks together, and encourages a high degree of modularity: recipes may be nested into other recipes, and configuration is cleanly separated from recipe logic. Tasks can be executed natively or in isolated environments using containerization technologies such as Apptainer. The container images are open-source and maintained through a companion package called cult-cargo. This enables the development of system-agnostic and fully reproducible workflows. Stimela2 facilitates the deployment of scalable, distributed workflows by interfacing with the Slurm scheduler and the Kubernetes API. The latter allows workflows to be readily deployed in the cloud. Previous papers in this series used Stimela2 as the underlying technology to run workflows on the AWS cloud. This paper presents an overview of Stimela2's design, architecture and use in the radio astronomy context.

Motivation & Objective

• To address the reproducibility crisis in radio astronomy by enabling fully auditable, version-controlled data reduction workflows.
• To bridge the gap between expert 'hero mode' data reduction and accessible, user-modifiable pipelines for non-expert users.
• To support seamless deployment of workflows across local, HPC, and cloud environments (e.g., AWS, Kubernetes) without vendor lock-in.
• To provide a modular, composable framework that simplifies integration of new tools and algorithms into standardized data reduction pipelines.
• To establish a certification model for workflows using versioned container images and cryptographic hashing to ensure computational and scientific reproducibility.

Proposed method

• Workflows are defined using concise, human-readable YAML recipes that describe task sequences and data flow.
• Atomic data reduction tasks are described via YAML 'cab' definitions specifying inputs, outputs, and execution context (e.g., Python, CASA, or binary).
• Tasks are executed natively or within isolated Apptainer containers, with container images managed via the open-source 'cult-cargo' package.
• The framework supports execution on local systems, Slurm-managed HPC clusters, and Kubernetes-based cloud environments for horizontal scaling.
• Reproducibility is enhanced by automatic generation of a dependencies file listing all software and image versions used in a workflow.
• A certification model is proposed, combining hashes of the YAML recipe, input data, and container images to produce a cryptographically verifiable workflow signature.

Experimental results

Research questions

• RQ1 How can data reduction workflows in radio astronomy be made both scalable and reproducible across diverse computing environments?
• RQ2 What architectural patterns enable modular, composable, and human-readable workflows without sacrificing performance or portability?
• RQ3 To what extent can containerization and versioned dependencies ensure computational and scientific reproducibility in complex data pipelines?
• RQ4 How can cloud-native orchestration (e.g., Kubernetes) be integrated into scientific workflows to support elastic, distributed execution?
• RQ5 What mechanisms can be established to certify workflows as reproducible, and how can they be adopted by the broader research community?

Key findings

• Stimela2 enables end-to-end data reduction workflows for radio astronomy using a modular, YAML-based recipe system that is both human-readable and machine-executable.
• The framework supports execution on local machines, HPC clusters via Slurm, and cloud environments via Kubernetes, enabling seamless scaling from small to large workloads.
• Containerization via Apptainer and the 'cult-cargo' package ensures that all dependencies are versioned and system-agnostic, significantly improving reproducibility.
• A certification model is proposed, where a cryptographically signed workflow signature can be generated from the recipe, input data, and container images, enabling verifiable reproducibility.
• The framework has been successfully used to deploy scalable workflows on AWS, demonstrating its viability for cloud-based radio astronomy processing.
• A public recipe competition is planned using real MeerKAT datasets hosted on AWS, aiming to enable quantitative, reproducible comparisons of calibration and imaging algorithms across the community.