[Paper Review] Scalable and Fault Tolerant Computation with the Sparse Grid Combination Technique

Brendan Harding, Markus Hegland+2 more|arXiv (Cornell University)|Apr 10, 2014

Distributed systems and fault tolerance13 references3 citations

TL;DR

This paper proposes a fault-tolerant extension of the sparse grid combination technique for scalable high-performance computing, enabling resilient, massively parallel solutions with minimal recovery overhead. By generalizing combination coefficients and using algorithm-based fault tolerance, the method reduces recovery time at the cost of negligible accuracy loss, outperforming traditional checkpoint-restart under frequent failures in 3D advection PDE simulations.

ABSTRACT

This paper continues to develop a fault tolerant extension of the sparse grid combination technique recently proposed in [B. Harding and M. Hegland, ANZIAM J., 54 (CTAC2012), pp. C394-C411]. The approach is novel for two reasons, first it provides several levels in which one can exploit parallelism leading towards massively parallel implementations, and second, it provides algorithm-based fault tolerance so that solutions can still be recovered if failures occur during computation. We present a generalisation of the combination technique from which the fault tolerant algorithm is a consequence. Using a model for the time between faults on each node of a high performance computer we provide bounds on the expected error for interpolation with this algorithm. Numerical experiments on the scalar advection PDE demonstrate that the algorithm is resilient to faults on a real application. It is observed that the trade-off of recovery time to decreased accuracy of the solution is suitably small. A comparison with traditional checkpoint-restart methods applied to the combination technique show that our approach is highly scalable with respect to the number of faults.

Motivation & Objective

• To address the challenge of run-time failures in exascale computing, where traditional checkpoint-restart mechanisms become infeasible due to high failure rates and long checkpoint times.
• To develop a fault-tolerant variant of the sparse grid combination technique that enables resilient, scalable computation on large-scale systems.
• To minimize recovery time and computational overhead while maintaining acceptable solution accuracy under multiple fault scenarios.
• To demonstrate scalability and resilience through numerical experiments on the 3D scalar advection PDE with simulated faults.

Proposed method

• Generalizes the sparse grid combination technique to allow dynamic re-computation of combination coefficients after node failures, enabling fault tolerance without full re-execution.
• Uses a probabilistic fault model to estimate the expected error in interpolated solutions under node failures, providing theoretical bounds on accuracy loss.
• Implements a multi-layered parallel architecture using MPI and OpenMP to exploit task, data, and component-level parallelism across distributed nodes.
• Applies algorithm-based fault tolerance (ABFT) by allowing small, bounded errors in solution reconstruction after failures, avoiding full checkpointing.
• Performs combination of component grid solutions only once at the end of computation, reducing communication overhead and improving scalability.
• Validates the approach using simulated faults in a 3D advection PDE solver, comparing recovery performance against local and global checkpointing.

Experimental results

Research questions

• RQ1 How can the sparse grid combination technique be generalized to support fault tolerance while maintaining computational efficiency?
• RQ2 What is the expected error in the solution when faults occur during computation, and can it be bounded mathematically?
• RQ3 How does the proposed algorithm-based fault tolerance compare to traditional checkpoint-restart methods in terms of recovery time and scalability under multiple faults?
• RQ4 What is the impact of fault frequency and distribution on solution accuracy and performance in a parallel HPC environment?

Key findings

• The proposed fault-tolerant combination technique reduces recovery time significantly compared to checkpoint-restart methods, especially under high fault rates.
• The expected error in the solution remains very small, with negligible impact on accuracy even after multiple simulated faults.
• The method scales well with increasing numbers of faults, outperforming both local and global checkpointing after only a few faults.
• The overhead of the fault-tolerant approach becomes negligible compared to recovery costs when multiple failures occur.
• The implementation achieves high scalability, with speedup approaching linear scaling on up to 60 threads in the absence of faults.
• The approach is robust in practice, with numerical experiments showing consistent resilience on a real 3D advection PDE application.