[Paper Review] RADICAL-Cybertools: Middleware Building Blocks for Scalable Science

Vivek Balasubramanian, Shantenu Jha+2 more|arXiv (Cornell University)|Apr 5, 2019

Scientific Computing and Data ManagementDecision Sciences13 references17 citations

TL;DR

RADICAL-Cybertools (RCT) provide a modular, composable middleware framework for scalable scientific computing, enabling high-performance execution of complex workloads across heterogeneous HPC and HTC infrastructures. By leveraging pilot-based task management and ensemble workflow abstractions, RCT support extreme-scale science in domains like biophysics and climate modeling, achieving hundreds of millions of core hours with low-code, portable, and interoperable execution across diverse platforms.

ABSTRACT

RADICAL-Cybertools (RCT) are a set of software systems that serve as middleware to develop efficient and effective tools for scientific computing. Specifically, RCT enable executing many-task applications at extreme scale and on a variety of computing infrastructures. RCT are building blocks, designed to work as stand-alone systems, integrated among themselves or integrated with third-party systems. RCT enables innovative science in multiple domains, including but not limited to biophysics, climate science and particle physics, consuming hundreds of millions of core hours. This paper provides an overview of RCT systems, their impact, and the architectural principles and software engineering underlying RCT

Motivation & Objective

• Address the challenge of supporting scalable, diverse, and evolving scientific workloads across heterogeneous computing infrastructures.
• Design sustainable, composable middleware building blocks that decouple functionality from infrastructure and enable reuse across scientific domains.
• Overcome limitations of monolithic middleware by enabling modular, composable, and interoperable systems that integrate with third-party tools.
• Support extreme-scale scientific applications requiring concurrent, heterogeneous, and dynamically orchestrated task execution across CPUs, GPUs, and accelerators.
• Enable domain scientists to focus on science rather than low-level workflow and resource management through high-level abstractions like pilot and ensemble workflows.

Proposed method

• Implement RADICAL-Pilot as a pilot-based execution model that abstracts computing resources into 'pilots' for efficient, direct scheduling of compute units (CUs) without batch queueing.
• Use RADICAL-Ensemble Toolkit (EnTK) to provide a high-level programming model for managing ensembles of tasks with arbitrary dependency and priority relationships.
• Leverage RADICAL-SAGA as a portable, Python-based interface to heterogeneous distributed resources, enabling interoperability across job schedulers, file transfer, and provisioning services.
• Apply a 'building blocks' architectural pattern based on modularity, stateless components, and dedicated communication meshes to ensure scalability, fault tolerance, and composability.
• Design components to be independently deployable and configurable, supporting both standalone and integrated use with third-party tools like Airflow, Oozie, and Spark Streaming.
• Integrate with production HPC and HTC systems (e.g., XSEDE, Blue Waters, DOE leadership machines) to validate performance and interoperability at scale.

Experimental results

Research questions

• RQ1 How can middleware be architected to support extreme-scale, heterogeneous, and dynamic scientific workloads across diverse computing infrastructures?
• RQ2 What architectural patterns enable sustainable, composable, and interoperable scientific middleware that can be maintained and extended over time?
• RQ3 To what extent can pilot-based and ensemble-based abstractions reduce the complexity of deploying and managing large-scale scientific workflows?
• RQ4 How do composable middleware building blocks improve developer productivity and reduce duplication of effort across scientific domains?
• RQ5 Can a small, agile team sustain and scale a high-impact middleware system across multiple concurrent scientific projects without monolithic development?

Key findings

• RCT have enabled execution of hundreds of millions of core hours across diverse scientific domains, including biophysics, climate science, and particle physics.
• The integration of HTBAC with RCT reduced time-to-solution for free-energy calculations by a factor of 2.5 in controlled experiments on real drug candidates.
• The AIMES project demonstrated that integrating task and resource metadata in scheduling decisions improves performance and scalability in federated HPC environments.
• ICEBERG leveraged both RADICAL-Pilot and EnTK within a single framework to support five distinct polar science use cases, with minimal API changes based on workload type.
• ExTASY and RepEx used EnTK to implement complex biomolecular sampling workflows with minimal engineering overhead, focusing on scientific innovation rather than workflow plumbing.
• The building blocks approach enabled sustainable development with only two active developers, supporting over ten concurrent scientific projects through composable, reusable, and interoperable components.