[Paper Review] INDIGO-DataCloud: A data and computing platform to facilitate seamless access to e-infrastructures.

Davide Salomoni, Isabel Campos+61 more|arXiv (Cornell University)|Nov 6, 2017

Distributed and Parallel Computing Systems35 references3 citations

TL;DR

INDIGO-DataCloud proposes a middleware platform that integrates public and private e-infrastructures—including EGI, EUDAT, and Helix Nebula—via AAI-compliant interfaces, enabling seamless, secure access to cloud, grid, and HPC resources. The system supports portable scientific applications using containerized execution across hybrid infrastructures, with open-source components already adopted in multiple scientific workflows.

ABSTRACT

This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications.

Motivation & Objective

• To address the challenge of fragmented access to diverse scientific e-infrastructures across public and private clouds, grids, and HPC clusters.
• To enable seamless integration of existing services from EGI, EUDAT, and Helix Nebula into a unified, interoperable platform.
• To support the deployment and execution of scientific applications across hybrid infrastructures using containerization and standardized authentication.
• To ensure trust and transparency in service provisioning through compliance with GEANT interfederation policies and AAI standards.
• To provide open-source middleware components that facilitate adoption across scientific communities and applications.

Proposed method

• Extending existing PaaS solutions to support hybrid resource federation across public and private e-infrastructures.
• Implementing AAI-compliant authentication and authorization mechanisms aligned with GEANT interfederation policies.
• Integrating services from EGI, EUDAT, and Helix Nebula into a unified middleware layer for transparent access.
• Enabling container-based execution of scientific applications on cloud, grid, and HPC environments.
• Designing a portable application interface that allows easy porting and execution across diverse infrastructures.
• Releasing all developed components as open-source software for community adoption and integration.

Experimental results

Research questions

• RQ1 How can scientific applications be seamlessly deployed and executed across heterogeneous e-infrastructures, including cloud, grid, and HPC systems?
• RQ2 What middleware architecture enables secure, interoperable access to distributed e-infrastructure services while maintaining trust and transparency?
• RQ3 To what extent can existing PaaS solutions be extended to support hybrid, multi-domain scientific computing environments?
• RQ4 How can AAI-compliant identity management be integrated across diverse e-infrastructures such as EGI, EUDAT, and Helix Nebula?
• RQ5 What mechanisms enable portable, containerized execution of scientific applications across public and private infrastructure domains?

Key findings

• The middleware successfully federated hybrid computing resources from EGI, EUDAT, and Helix Nebula into a unified access layer.
• The platform enables transparent and secure access to services through AAI-compliant authentication, ensuring trust across domains.
• Scientific applications can be easily written, ported, and executed across cloud, grid, and HPC infrastructures using containerization.
• The developed components are freely available as open-source software and are already integrated into multiple scientific applications.
• The solution supports interoperability across existing e-infrastructures while complying with GEANT's interfederation policies.
• The system demonstrates practical feasibility and adoption potential through real-world integration in scientific workflows.