[Paper Review] A Flexible High Demand Storage System for MAGIC-I and MAGIC-II using GFS

E. Carmona, J. A. Coarasa+1 more|arXiv (Cornell University)|Jul 6, 2009

Astrophysics and Cosmic Phenomena3 references3 citations

TL;DR

This paper presents a scalable, reliable, and flexible high-performance storage system for the MAGIC-I and MAGIC-II Cherenkov telescopes using the Global File System (GFS) on a Red Hat Cluster. The system supports sustained DAQ write rates above 1.2 kHz, enables concurrent data processing (compression, taping, on-line analysis), and ensures fault tolerance through RAID5, multipath I/O, and automatic fencing, achieving 1+ TB of nightly data storage with minimal downtime and seamless scalability.

ABSTRACT

MAGIC-I is currently the Imaging Cherenkov Telescope with the worldwide largest reflector currently in operation. The initially achieved low trigger threshold of 60 GeV has been further reduced by means of a novel trigger that allows the telescope to record gamma ray showers down to 25 GeV. The high trigger rate combined with the 2 GHz signal sampling rate results in large data volumes that can reach 1 TByte per night for MAGIC-I and even more with the second MAGIC telescope coming soon into operation. To deal with the large storage requirements of MAGIC-I and MAGIC-II, we have installed the distributed file system GFS and a cluster of computers with concurrent access to the same shared storage units. The system can not only handle a sustained DAQ write rate above 1.2 kHz for MAGIC-I, but also allows other nodes to perform simultaneous concurrent access to the data on the shared storage units. Various simultaneous tasks can be used at any time, in parallel with data taking, including data compression, taping, on-line analysis, calibration and analysis of the data. The system is designed to quickly recover after the failure of one node in the cluster and to be easily extended as more nodes or storage units are required in the future.

Motivation & Objective

• Address the high data volume and I/O demands from MAGIC telescopes, which generate up to 60 MB/s during Sum Trigger operations.
• Provide a consistent, shared file system across multiple nodes to support concurrent data processing tasks like compression, taping, and on-line analysis.
• Ensure high reliability through RAID5, multipath I/O, and automatic fencing of failed nodes.
• Enable dynamic resource allocation and system scalability to accommodate the upcoming MAGIC-II telescope with 1039 pixels and doubled data rates.
• Design a system that supports rapid recovery from node or storage failures without data corruption.

Proposed method

• Deployed a Red Hat Cluster Suite with seven nodes, each having direct read/write access to shared Fibre Channel-attached RAID storage units.
• Implemented the Global File System (GFS) to provide a consistent, shared file system view across all cluster nodes.
• Used a dedicated 1 Gbps Ethernet network for cluster messaging (CMAN) and a separate Fibre Channel network for storage I/O.
• Configured RAID5 arrays with spare disks to ensure data integrity and automatic failover upon disk failure.
• Enabled multipath I/O using the multipath daemon to maintain connectivity across redundant Fibre Channel paths.
• Employed the CMAN and fence services to automatically detect and isolate failed nodes, preventing data corruption by blocking access to shared storage.

Experimental results

Research questions

• RQ1 How can a high-performance, scalable, and reliable storage system be designed to handle 1+ TB of nightly data from Cherenkov telescopes with high trigger rates?
• RQ2 What mechanisms ensure data consistency and fault tolerance when multiple nodes access shared storage simultaneously?
• RQ3 How can the system support concurrent data processing tasks (e.g., compression, taping, on-line analysis) without I/O bottlenecks?
• RQ4 What strategies enable seamless scalability when upgrading to a second telescope with higher data rates?
• RQ5 How can node and storage failures be detected and isolated automatically to prevent data corruption?

Key findings

• The system sustained DAQ write rates above 1.2 kHz for MAGIC-I, handling up to 60 MB/s during high-trigger-rate observations.
• The system achieved a data volume of over 1 TB per night under high-demand conditions, with raw data reaching 60 MB/s during Sum Trigger operations.
• The GFS-based cluster enabled concurrent access to data for compression, taping, on-line analysis, and calibration, with I/O performance matching the DAQ write speed.
• The system recovered from node failures in a few seconds using automatic fencing and CMAN, preventing data corruption.
• The storage architecture supported dynamic scaling, with less than 30% of Fibre Channel switch ports currently in use, allowing for future expansion of up to 100 nodes.
• The use of RAID5 with automatic spare disk replacement and multipath I/O ensured high reliability, with no data loss during disk or path failures.