[Paper Review] A Monitoring System for the BaBar INFN Computing Cluster

Moreno Marzolla, V. Melloni|ArXiv.org|May 29, 2003

Distributed and Parallel Computing Systems4 references3 citations

TL;DR

PerfMC is a lightweight, XML-driven monitoring system for large computing clusters that uses SNMP to collect performance data from diverse devices, including PCs, switches, and tape libraries. It embeds a web server to generate real-time graphs and customizable HTML/XML status pages via XSLT, achieving low overhead and scalability for clusters of up to 200 dual-CPU machines with polling intervals of ~10 seconds.

ABSTRACT

Monitoring large clusters is a challenging problem. It is necessary to observe a large quantity of devices with a reasonably short delay between consecutive observations. The set of monitored devices may include PCs, network switches, tape libraries and other equipments. The monitoring activity should not impact the performances of the system. In this paper we present PerfMC, a monitoring system for large clusters. PerfMC is driven by an XML configuration file, and uses the Simple Network Management Protocol (SNMP) for data collection. SNMP is a standard protocol implemented by many networked equipments, so the tool can be used to monitor a wide range of devices. System administrators can display informations on the status of each device by connecting to a WEB server embedded in PerfMC. The WEB server can produce graphs showing the value of different monitored quantities as a function of time; it can also produce arbitrary XML pages by applying XSL Transformations to an internal XML representation of the cluster's status. XSL Transformations may be used to produce HTML pages which can be displayed by ordinary WEB browsers. PerfMC aims at being relatively easy to configure and operate, and highly efficient. It is currently being used to monitor the Italian Reprocessing farm for the BaBar experiment, which is made of about 200 dual-CPU Linux machines.

Motivation & Objective

• Address the challenge of monitoring large, high-performance computing clusters with hundreds of nodes efficiently and non-intrusively.
• Overcome limitations of existing monitoring tools that are often closed-source, hard to configure, or unsuitable for high-speed LAN-based clusters.
• Provide a flexible, extensible monitoring solution that supports heterogeneous devices through standard protocols like SNMP.
• Enable real-time visualization and reporting of cluster status using web-based, XSLT-transformed XML outputs.
• Ensure low network and system overhead to avoid impacting cluster performance, especially at moderate polling frequencies.

Proposed method

• Uses an XML configuration file to define monitoring targets and parameters, enabling easy setup and customization.
• Employs SNMPv2 bulk requests to efficiently retrieve multiple MIB variables from each device in a single round-trip, minimizing network load.
• Embeds a lightweight web server within PerfMC to serve dynamic HTML pages generated via XSLT transformations of internal XML status data.
• Stores performance metrics in Round Robin Database (RRD) files for time-series analysis and graph generation.
• Supports extensibility by allowing custom MIBs through the Net-SNMP framework, enabling monitoring of application-level metrics.
• Designs for horizontal scalability by supporting a hierarchical architecture with multiple monitors, each managing a subset of the cluster.

Experimental results

Research questions

• RQ1 How can a monitoring system be designed to efficiently scale across large, homogeneous computing clusters with minimal performance impact?
• RQ2 To what extent can standard, open protocols like SNMP and extensible data formats like XML and XSLT be leveraged to build a flexible and maintainable monitoring infrastructure?
• RQ3 What architectural patterns enable low-overhead, real-time monitoring of diverse devices (e.g., servers, switches, tape libraries) in a high-speed LAN environment?
• RQ4 How can monitoring data be visualized and exposed through standard web technologies without requiring custom client software?
• RQ5 What mechanisms can be used to extend the system to support fault tolerance and hierarchical monitoring in very large clusters?

Key findings

• PerfMC achieves low network and system overhead, with negligible impact on monitored devices and network when polling at ~10-second intervals.
• The use of SNMPv2 bulk operations significantly reduces the number of packets required to collect data from multiple MIB variables, improving efficiency.
• The integration of XSLT transformations enables dynamic, customizable web-based status reports and dashboards without requiring custom web development.
• The system demonstrates scalability for clusters of up to 200 dual-CPU machines, with no observed performance bottlenecks in the current prototype.
• The authors identify RRDTool’s update performance as a potential scalability bottleneck and propose partitioning the cluster across multiple monitors as a long-term solution.
• An alarm system based on threshold monitoring and RMON-compliant SNMP agents is planned, with support for both MIB variable thresholds and host unresponsiveness detection.