[Paper Review] A web portal for hydrodynamical, cosmological simulations

Antonio Ragagnin, Klaus Dolag+6 more|arXiv (Cornell University)|Dec 19, 2016

Computational Physics and Python Applications29 references4 citations

TL;DR

This paper presents a multi-layered web portal that enables researchers to access, query, and analyze large-scale cosmological hydrodynamical simulations via remote high-performance computing. The system supports advanced object selection, real-time data processing, and specialized analysis tools—such as virtual X-ray observations and 2D map generation—demonstrating a scalable, secure infrastructure for sharing complex astrophysical data with the broader scientific community.

ABSTRACT

This article describes a data center hosting a web portal for accessing and sharing the output of large, cosmological, hydro-dynamical simulations with a broad scientific community. It also allows users to receive related scientific data products by directly processing the raw simulation data on a remote computing cluster. The data center has a multi-layer structure: a web portal, a job control layer, a computing cluster and a HPC storage system. The outer layer enables users to choose an object from the simulations. Objects can be selected by visually inspecting 2D maps of the simulation data, by performing highly compounded and elaborated queries or graphically by plotting arbitrary combinations of properties. The user can run analysis tools on a chosen object. These services allow users to run analysis tools on the raw simulation data. The job control layer is responsible for handling and performing the analysis jobs, which are executed on a computing cluster. The innermost layer is formed by a HPC storage system which hosts the large, raw simulation data. The following services are available for the users: (I) {\sc ClusterInspect} visualizes properties of member galaxies of a selected galaxy cluster; (II) {\sc SimCut} returns the raw data of a sub-volume around a selected object from a simulation, containing all the original, hydro-dynamical quantities; (III) {\sc Smac} creates idealised 2D maps of various, physical quantities and observables of a selected object; (IV) {\sc Phox} generates virtual X-ray observations with specifications of various current and upcoming instruments.

Motivation & Objective

• To address the challenge of sharing and analyzing massive, complex cosmological simulation data across a broad scientific community.
• To provide a scalable, secure, and user-friendly infrastructure for accessing raw simulation outputs and derived scientific products.
• To enable researchers to perform on-demand analysis on remote HPC clusters without downloading terabyte-scale datasets.
• To support advanced scientific workflows, including virtual instrument simulations and multi-parameter object selection.
• To ensure long-term data accessibility and extensibility for upcoming surveys and evolving simulation datasets.

Proposed method

• The portal employs a four-layer architecture: web interface, job control layer, HPC computing cluster, and HPC storage system.
• Users select objects via visual inspection of 2D maps, scatter plots, or complex SQL-like queries on Subfind substructure data.
• Analysis jobs are submitted through the portal and executed on a remote HPC cluster, which directly reads raw simulation data from the storage system.
• Four core services are implemented: ClusterInspect (galaxy cluster member analysis), SimCut (sub-volume extraction), Smac (2D physical map generation), and Phox (virtual X-ray observations).
• Each service supports customizable parameters, such as projection axes, physical quantities, and instrument models (e.g., XMM, eROSITA, Athena).
• The system ensures security and performance through isolated job execution and direct data access, avoiding local data transfer.

Experimental results

Research questions

• RQ1 How can large-scale cosmological hydrodynamical simulations be made accessible and analyzable by a broad scientific community without requiring local storage of terabyte-scale datasets?
• RQ2 What architectural design enables efficient, secure, and scalable remote analysis of complex simulation outputs on HPC infrastructure?
• RQ3 How can users perform advanced, compound queries on simulation substructures (e.g., galaxy clusters by dynamical state or compactness) through an intuitive interface?
• RQ4 To what extent can virtual instrument simulations (e.g., X-ray observations) be integrated into a web-based analysis portal for cosmological data?
• RQ5 What mechanisms ensure long-term data availability and extensibility for evolving simulation projects and future astronomical surveys?

Key findings

• The portal successfully enables users to select and analyze cosmological objects—such as galaxy clusters—via visual, parametric, or query-based methods without local data storage.
• The system supports the generation of 2D physical maps (Smac) and virtual X-ray observations (Phox) with instrument-specific response functions, including for upcoming missions like Athena.
• SimCut allows on-demand extraction of raw simulation data (e.g., particle data) for any selected sub-volume, preserving full hydrodynamical quantities.
• The portal has been deployed for the Magneticum Pathfinder simulations, which cover $2 \times 4536^3$ resolution elements and reach up to several hundred terabytes of raw data.
• The infrastructure is secured for five years and is designed to be extensible, with source code available on request for adaptation to other simulation projects.
• The system enables reproducible, remote analysis of complex astrophysical data, significantly lowering barriers for researchers not equipped with local HPC resources.