[Paper Review] Building A High Performance Parallel File System Using Grid Datafarm and ROOT I/O

Y. Morita, Hiroyuki Satō+6 more|ArXiv.org|Jun 14, 2003

Distributed and Parallel Computing Systems6 references3 citations

TL;DR

This paper presents a high-performance parallel file system built on Grid Datafarm (Gfarm) and ROOT I/O for managing petabyte-scale data in high-energy nuclear physics (HENP) experiments. By leveraging Gfarm’s distributed storage and replication capabilities with ROOT’s efficient I/O for histogram and event data, the system achieved over 2.3 Gbps data transfer rates across seven clusters in the US and Japan, demonstrating scalable, parallel data management for large-scale detector simulations.

ABSTRACT

Sheer amount of petabyte scale data foreseen in the LHC experiments require a careful consideration of the persistency design and the system design in the world-wide distributed computing. Event parallelism of the HENP data analysis enables us to take maximum advantage of the high performance cluster computing and networking when we keep the parallelism both in the data processing phase, in the data management phase, and in the data transfer phase. A modular architecture of FADS/ Goofy, a versatile detector simulation framework for Geant4, enables an easy choice of plug-in facilities for persistency technologies such as Objectivity/DB and ROOT I/O. The framework is designed to work naturally with the parallel file system of Grid Datafarm (Gfarm). FADS/Goofy is proven to generate 10^6 Geant4-simulated Atlas Mockup events using a 512 CPU PC cluster. The data in ROOT I/O files is replicated using Gfarm file system. The histogram information is collected from the distributed ROOT files. During the data replication it has been demonstrated to achieve more than 2.3 Gbps data transfer rate between the PC clusters over seven participating PC clusters in the United States and in Japan.

Motivation & Objective

• To address the challenge of managing petabyte-scale data generated by LHC experiments in a distributed, high-performance computing environment.
• To enable end-to-end event parallelism in data processing, management, and transfer for optimal cluster utilization.
• To integrate ROOT I/O with the Gfarm parallel file system to support efficient, scalable data persistency in distributed detector simulations.
• To demonstrate high-throughput data replication and histogram collection across geographically distributed PC clusters.

Proposed method

• Utilized the FADS/Goofy framework, a modular detector simulation system based on Geant4, to support flexible plug-in selection for data persistency.
• Employed Gfarm as the underlying parallel file system to manage data distribution, replication, and access across multiple clusters.
• Used ROOT I/O to store and retrieve event and histogram data efficiently, supporting high I/O throughput.
• Enabled data replication across seven clusters in the US and Japan using Gfarm’s distributed replication mechanism.
• Collected histogram information from distributed ROOT files to support global analysis without centralizing data.
• Achieved high performance by maintaining data parallelism throughout processing, management, and transfer phases.

Experimental results

Research questions

• RQ1 How can a scalable, high-performance parallel file system be architected to manage petabyte-scale HENP data across distributed computing infrastructures?
• RQ2 What is the achievable data transfer rate when replicating large-scale ROOT I/O files across geographically dispersed clusters using Gfarm?
• RQ3 Can a modular simulation framework like FADS/Goofy effectively integrate with both Gfarm and ROOT I/O to support high-throughput data management?
• RQ4 How can event-level parallelism be preserved across data processing, management, and transfer phases in a distributed environment?
• RQ5 What performance gains are realized by combining Gfarm’s replication with ROOT’s I/O optimization in a multi-cluster setting?

Key findings

• The system successfully generated 1 million Geant4-simulated Atlas Mockup events using a 512-CPU PC cluster, validating scalability and performance.
• Data replication across seven clusters in the US and Japan achieved a sustained transfer rate exceeding 2.3 Gbps, demonstrating high network and I/O efficiency.
• Histogram data was efficiently collected from distributed ROOT files without requiring data centralization, enabling scalable analysis.
• The integration of FADS/Goofy with Gfarm and ROOT I/O enabled seamless, modular data persistency with minimal performance overhead.
• The architecture maintained end-to-end event parallelism, maximizing utilization of high-performance clusters and networks.
• The solution proved effective for large-scale, distributed HENP data workloads, supporting future petabyte-scale data management needs.