[Paper Review] The use of Ethernet in the DataFlow of the ATLAS Trigger & DAQ

S. Stancu, Bob Dobinson+3 more|arXiv (Cornell University)|May 30, 2003

Distributed and Parallel Computing Systems2 references7 citations

TL;DR

This paper evaluates Ethernet's suitability for the ATLAS Trigger and Data Acquisition (TDAQ) system, proposing a high-performance network architecture using Gigabit Ethernet with critical features like flow control, VLANs, QoS, and trunking. It demonstrates that with proper configuration, Ethernet can reliably handle the system's high data rates and real-time constraints, achieving stable performance under load with effective traffic shaping and flow control mechanisms.

ABSTRACT

The article analyzes a proposed network topology for the ATLAS DAQ DataFlow, and identifies the Ethernet features required for a proper operation of the network: MAC address table size, switch performance in terms of throughput and latency, the use of Flow Control, Virtual LANs and Quality of Service. We investigate these features on some Ethernet switches, and conclude on their usefulness for the ATLAS DataFlow network.

Motivation & Objective

• To assess Ethernet's viability as the backbone for the ATLAS TDAQ DataFlow system.
• To identify essential Ethernet features required for reliable, low-latency operation in a high-throughput, real-time environment.
• To evaluate switch capabilities—throughput, latency, MAC table size, flow control, VLANs, QoS—on real hardware for ATLAS use cases.
• To ensure network stability and fault tolerance through proper topology design and traffic management.
• To validate that commodity Ethernet hardware can meet the stringent performance and reliability demands of LHC-scale data acquisition.

Proposed method

• Proposes a network architecture using Gigabit Ethernet for PCs and Fast Ethernet for ROBs, interconnected via GE optical links.
• Employs a three-tiered message flow: L2SV load-balances RoI requests to L2PUs, which perform incremental analysis and report decisions to DFM.
• Uses unicast messaging for full traffic control, with multicast only for low-rate 'clear' messages to ROBs.
• Implements traffic shaping via request-response patterns to prevent buffer overflow and enable flow control.
• Tests flow control on both send and receive sides, identifying limitations in user-level application responsiveness.
• Evaluates trunking (IEEE 802.3ad) for bandwidth aggregation, analyzing load balancing behavior under real traffic patterns.

Experimental results

Research questions

• RQ1 Can standard Ethernet switches meet the performance and reliability requirements of the ATLAS TDAQ DataFlow network?
• RQ2 How effective is flow control in managing data rates between high-speed NICs and processing nodes?
• RQ3 What impact does trunking (LAG) have on load distribution and network performance in a large-scale, multi-node system?
• RQ4 How do QoS, VLANs, and switch MAC table size affect network stability and scalability in the ATLAS context?
• RQ5 Can commodity Ethernet hardware, with proper configuration, support real-time data acquisition at 80 TB/s raw data rates?

Key findings

• Ethernet is suitable for ATLAS TDAQ due to its high bandwidth, multi-vendor support, and evolutionary upgrade path to 10 Gbps.
• Switches must support large MAC address tables (≥1600 entries) and low latency to handle the ROB and L2PU scale.
• Flow control effectively reduces sending rates when buffers fill, but is ineffective on the receive side if user-level applications do not empty sockets promptly.
• Trunking can increase bandwidth, but random frame distribution across links may lead to uneven load; however, with many connections, this imbalance is statistically mitigated.
• A dual Pentium 4 (2.4 GHz) PC can sustain ~70 MB/s reception in request-reply scenarios, confirming PC capability for Gigabit Ethernet.
• The use of unicast messaging enables full control over traffic patterns, while multicast is limited to low-rate clear messages, ensuring switch compatibility.