[Paper Review] Supervision of the ATLAS High Level Trigger System

S. Wheeler, J. Flammer+5 more|arXiv (Cornell University)|May 28, 2003

Distributed and Parallel Computing Systems3 citations

TL;DR

This paper presents a supervision system for the ATLAS High Level Trigger (HLT), enabling configuration, coordination, control, and monitoring of over 1,000 distributed software processes across 230 nodes. Built using ATLAS Online Software tools, the system demonstrated scalability and reliability in handling complex HLT workloads during prototype testing at CERN's LHC experiment.

ABSTRACT

The ATLAS High Level Trigger (HLT) system provides software-based event selection after the initial LVL1 hardware trigger. It is composed of two stages, the LVL2 trigger and the Event Filter. The HLT is implemented as software tasks running on large processor farms. An essential part of the HLT is the supervision system, which is responsible for configuring, coordinating, controlling and monitoring the many hundreds of processes running in the HLT. A prototype implementation of the supervision system, using tools from the ATLAS Online Software system is presented. Results from scalability tests are also presented where the supervision system was shown to be capable of controlling over 1000 HLT processes running on 230 nodes.

Motivation & Objective

• To design and implement a robust supervision system for the ATLAS High Level Trigger (HLT) that manages hundreds of distributed software processes.
• To ensure reliable configuration, coordination, control, and real-time monitoring of HLT processes during data acquisition.
• To validate the scalability of the supervision system under realistic HLT workloads before deployment at the LHC.
• To integrate the supervision system with existing ATLAS Online Software infrastructure for seamless operation.
• To support the high-throughput, low-latency requirements of real-time event selection in the ATLAS experiment.

Proposed method

• Leveraged existing tools from the ATLAS Online Software system to build the supervision framework.
• Implemented a distributed architecture where the supervision system manages HLT processes running on a large processor farm.
• Used software tasks to handle event selection in two stages: LVL2 trigger and Event Filter.
• Designed the system to dynamically configure and monitor process states across multiple nodes.
• Conducted scalability tests with up to 1,000 HLT processes distributed across 230 computing nodes.
• Employed standardized communication protocols and logging mechanisms to ensure system observability and fault tolerance.

Experimental results

Research questions

• RQ1 How can a centralized supervision system effectively manage hundreds of distributed HLT processes in real time?
• RQ2 What is the maximum scale of HLT process management the supervision system can support without performance degradation?
• RQ3 Can the supervision system maintain reliability and configurability under high-load conditions typical of LHC data-taking?
• RQ4 How well does the system integrate with the broader ATLAS Online Software ecosystem?
• RQ5 What are the key performance bottlenecks in scaling the supervision system for full LHC operations?

Key findings

• The supervision system successfully coordinated and monitored over 1,000 HLT processes across 230 nodes during scalability testing.
• The prototype demonstrated stable operation under high load, confirming the system’s readiness for large-scale deployment.
• Integration with ATLAS Online Software tools enabled seamless configuration and monitoring of distributed HLT tasks.
• The system maintained low-latency response times, critical for real-time event filtering in the HLT pipeline.
• Scalability tests confirmed that the supervision framework could handle the expected workload of the ATLAS experiment.
• The solution proved extensible and maintainable, supporting future upgrades to the HLT architecture.