[Paper Review] 'Mutual Watch-dog Networking': Distributed Awareness of Faults and Critical Events in Petascale/Exascale systems

Roberto Ammendola, A. Biagioni+8 more|arXiv (Cornell University)|Jul 1, 2013

Distributed systems and fault tolerance6 references3 citations

TL;DR

This paper proposes 'Mutual Watch-dog Networking' (LO|FA|MO), a distributed fault-awareness framework for petascale and exascale systems that enables real-time detection and propagation of faults via redundant, low-speed diagnostic networks. By combining hardware monitoring, watchdog mechanisms, and hierarchical fault signaling, the system achieves early fault detection and resilient response, reducing system downtime in large-scale, high-complexity architectures.

ABSTRACT

Many tile systems require techniques to be applied to increase components resilience and control the FIT (Failures In Time) rate. When scaling to peta- exa-scale systems the FIT rate may become unacceptable due to component numerosity, requiring more systemic countermeasures. Thus, the ability to be fault aware, i.e. to detect and collect information about fault and critical events, is a necessary feature that large scale distributed architectures must provide in order to apply systemic fault tolerance techniques. In this context, the LO|FA|MO approach is a way to obtain systemic fault awareness, by implementing a mutual watchdog mechanism and guaranteeing fault detection in a no-single-point-of-failure fashion. This document contains specification and implementation details about this approach, in the shape of a technical report.

Motivation & Objective

• To address the escalating failure rates in petascale and exascale systems due to massive component counts and high complexity.
• To reduce system downtime by enabling early detection and propagation of faults before they cause catastrophic failures.
• To design a fault-aware infrastructure that complements proactive mitigation strategies like dynamic voltage scaling and process migration.
• To ensure fault resilience in systems with high component failure rates by introducing a redundant, low-speed diagnostic network independent of the main data network.
• To enable fault detection and signaling even when the primary network fails, using a fault-tolerant auxiliary network for diagnostic information propagation.

Proposed method

• The LO|FA|MO framework uses a mutual watchdog mechanism between the DNP (Distributing Network Processor) and a host processing element to detect faults via periodic status checks.
• A dedicated, low-speed, highly reliable diagnostic network (Service Network) is used to propagate fault information even if the main 3D toroidal network fails.
• Hardware monitors track temperature, voltage, current, BER counters, and internal exception registers to detect fault symptoms in real time.
• Fault status is signaled through registers: DNP Local/Global Status, Host Watchdog register, and DNP Watchdog register fields indicating faulty neighbors.
• The Host Fault Manager periodically checks the DNP’s status and propagates fault events to higher-level supervisors via the Service Network.
• In case of host or bus failure, the DNP detects watchdog timeout and signals neighboring nodes via the DNP Watchdog register, enabling global fault awareness.

Experimental results

Research questions

• RQ1 How can fault awareness be effectively distributed across petascale and exascale systems with tens of thousands of components?
• RQ2 What mechanisms enable reliable fault detection and signaling when the primary communication network is compromised?
• RQ3 How can mutual watchdogs between DNP and host processors improve fault detection latency and system resilience?
• RQ4 What role does a low-speed, dedicated diagnostic network play in maintaining fault awareness during main network failures?
• RQ5 How can fault information be propagated hierarchically to enable coordinated system-level responses?

Key findings

• The LO|FA|MO framework enables detection of DNP core faults, including routing logic errors and RDMA engine failures, through exception register monitoring and status flagging.
• DNP core meltdown faults are detected when the DNP fails to update its periodic status register, triggering watchdog timeout detection by the host.
• Temperature and voltage threshold violations are detected by monitoring dedicated registers and signaled via the DNP Local/Global Status register.
• Host-side peripheral faults are detected by the Host Local Fault Manager and communicated via the Host Watchdog register, which triggers diagnostic packet generation in neighboring DNPs.
• Total host breakdown or bus failure is detected by the DNP as a watchdog timeout, leading to fault propagation through the DNP Watchdog register and global awareness.
• The system maintains fault awareness even when the main 3D toroidal network fails, by relying on the resilient, independent Service Network for diagnostic information delivery.