[Paper Review] FirecREST: RESTful API on Cray XC systems

Felipe A. Cruz, Maxime Martinasso|arXiv (Cornell University)|Nov 29, 2019

Scientific Computing and Data Management8 references4 citations

TL;DR

FirecREST is a RESTful API infrastructure that enables science gateways to programmatically access and manage HPC resources—such as job submission, monitoring, and data transfer—on Cray XC systems at the Swiss National Supercomputing Centre (CSCS). By exposing core HPC services via standardized, stateless REST endpoints with integrated authentication, it simplifies secure integration for scientific platforms while supporting asynchronous workflows through a dedicated task management microservice.

ABSTRACT

As science gateways are becoming an increasingly popular digital interface for scientific communities, it is also becoming increasingly important for High-Performance Computing centers to provide a modern Web-enabled APIs. With such interface, science gateways can easily integrate access to HPC center resources. This work presents the FirecREST API, a RESTful Web API infrastructure that allows scientific communities to access the various integrated resources and services available from the Cray XC systems at the Swiss National Supercomputing Centre. FirecREST requirements are derived from use cases described in this work.

Motivation & Objective

• To address the growing need for standardized, secure, and interoperable Web APIs to integrate science gateways with HPC resources at national supercomputing centers.
• To enable scientific communities to access and manage HPC workloads and data on Cray XC systems through a modern, RESTful interface.
• To reduce the complexity and maintenance burden for gateway developers by providing a single, reusable API layer for HPC access.
• To support asynchronous operations and scalable service composition through a microservices-architected API with task tracking.
• To ensure compatibility with modern authentication standards by integrating with CSCS’s existing authorization and authentication infrastructure.

Proposed method

• FirecREST implements a microservices-architected RESTful API using OpenAPI specification to define endpoints, operations, parameters, and authentication methods.
• Core services include job management (submit, monitor), data transfer across filesystems, and task tracking for asynchronous operations.
• The API integrates with CSCS’s centralized authentication and authorization system, supporting external identity providers for secure access.
• Each microservice (e.g., job, storage, tasks) exposes standardized endpoints with stateless operations, enabling modular and scalable access to HPC resources.
• Asynchronous operations are managed via a dedicated task microservice that tracks progress and allows clients to poll for status updates.
• The system is designed to be generic and portable, with potential extension to non-Cray HPC systems by interfacing with batch schedulers and storage technologies.

Experimental results

Research questions

• RQ1 How can a standardized, secure, and scalable RESTful API be designed to enable science gateways to access HPC resources on Cray XC systems?
• RQ2 What architectural patterns and service abstractions are required to support job submission, monitoring, and data transfer via a Web API in an HPC environment?
• RQ3 How can authentication and authorization be effectively integrated into a RESTful HPC API to support multi-tenant and external identity provider access?
• RQ4 What mechanisms are needed to support asynchronous workflows, such as long-running data transfers or job execution, in a stateless API design?
• RQ5 To what extent can a generic HPC API be decoupled from specific hardware or software stacks to enable reuse across different HPC centers?

Key findings

• FirecREST successfully exposes core HPC functionalities—job submission, monitoring, and data transfer—through a standardized RESTful API with consistent, stateless endpoints.
• The integration with CSCS’s existing authentication infrastructure enables secure, identity-aware access to HPC resources without requiring changes to user workflows.
• The use of a dedicated task microservice enables effective tracking of asynchronous operations, such as data transfers, allowing clients to monitor progress without blocking.
• The OpenAPI specification enables machine-readable documentation, automated client library generation, and improved interoperability across gateway platforms.
• The architecture supports multiple use cases, including RENKU for data science, Materials Cloud with AiiDA, and Jupyter-based interactive services, demonstrating broad applicability.
• FirecREST is designed to be extensible and generic, with future support planned for node reservation and data transformation services, indicating long-term adaptability.