[Paper Review] Algorithmic Based Fault Tolerance Applied to High Performance Computing

George Bosilca, Rémi Delmas+2 more|ArXiv.org|Jun 19, 2008

Distributed systems and fault tolerance1 references20 citations

TL;DR

This paper presents a novel, scalable algorithmic-based fault tolerance (ABFT) technique for high-performance computing, extending ABFT to parallel distributed matrix-matrix multiplication (PDGEMM). It enables on-the-fly detection and correction of process failures and bit-flip errors with only 12% overhead at 484 processors, achieving 1.4 TFLOPS and 65% of peak performance while reducing fault tolerance overhead as processor count increases.

ABSTRACT

We present a new approach to fault tolerance for High Performance Computing system. Our approach is based on a careful adaptation of the Algorithmic Based Fault Tolerance technique (Huang and Abraham, 1984) to the need of parallel distributed computation. We obtain a strongly scalable mechanism for fault tolerance. We can also detect and correct errors (bit-flip) on the fly of a computation. To assess the viability of our approach, we have developed a fault tolerant matrix-matrix multiplication subroutine and we propose some models to predict its running time. Our parallel fault-tolerant matrix-matrix multiplication scores 1.4 TFLOPS on 484 processors (cluster jacquard.nersc.gov) and returns a correct result while one process failure has happened. This represents 65% of the machine peak efficiency and less than 12% overhead with respect to the fastest failure-free implementation. We predict (and have observed) that, as we increase the processor count, the overhead of the fault tolerance drops significantly.

Motivation & Objective

• Address the scalability limitations of traditional checkpointing in large-scale HPC systems with high failure rates.
• Develop a fault-tolerant linear algebra kernel that maintains performance as processor count increases.
• Enable on-the-fly detection and correction of both process failures and bit-flip errors during computation.
• Minimize fault tolerance overhead by leveraging algorithmic encoding and parallelism, avoiding expensive I/O.
• Demonstrate viability of ABFT in a distributed, message-passing environment like ScaLAPACK.

Proposed method

• Adapted Huang and Abraham's original ABFT technique to work in a parallel, distributed-memory environment using ScaLAPACK's PDGEMM.
• Encoded matrix data using checksum schemes (e.g., parity vectors) to enable error detection and correction during computation.
• Redesigned the matrix-matrix multiplication algorithm to operate on encoded data, allowing fault detection and correction without restarting.
• Integrated the fault-tolerant kernel into the ABFT BLAS library, enabling transparent error recovery during execution.
• Used a predictive performance model with two parameters (α and γ) to estimate execution time and overhead across different processor counts.
• Conducted weak and strong scalability experiments on a 484-processor cluster to evaluate performance and overhead under failure conditions.

Experimental results

Research questions

• RQ1 Can ABFT be effectively extended to large-scale, distributed-memory HPC systems to achieve fault tolerance with low overhead?
• RQ2 Does the fault tolerance overhead of ABFT decrease as the number of processors increases, even with a fixed problem size?
• RQ3 Can the proposed method detect and correct both process failures and bit-flip errors in real time during matrix-matrix multiplication?
• RQ4 How does the performance of the fault-tolerant PDGEMM compare to the standard PBLAS PDGEMM in weak and strong scaling scenarios?
• RQ5 To what extent can the overhead of fault tolerance be predicted using a simple model with only two parameters?

Key findings

• The fault-tolerant PDGEMM achieved 1.4 TFLOPS on 484 processors (cluster jacquard.nersc.gov), reaching 65% of the machine's peak performance.
• With one process failure, the algorithm correctly returned the result, demonstrating effective on-the-fly recovery.
• The fault tolerance overhead was less than 12% compared to the fastest failure-free implementation at 484 processors.
• As the number of processors increased, the fault tolerance overhead decreased significantly, dropping to 114.7% at 484 processors in strong scaling.
• The performance model with two parameters (α and γ) predicted experimental results within a few percent across 48 experiments.
• In strong scaling, with fixed problem size, the overhead of fault tolerance converged to zero as processor count increased, demonstrating inherent scalability.