[Paper Review] Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs

Jack Kosaian, K. V. Rashmi|arXiv (Cornell University)|Apr 19, 2021

Advanced Neural Network ApplicationsComputer Science69 references43 citations

TL;DR

This paper proposes intensity-guided ABFT, an adaptive fault tolerance framework for neural network inference on GPUs that dynamically selects between traditional and thread-level algorithm-based fault tolerance (ABFT) based on each layer's arithmetic intensity. By exploiting underutilized compute resources in bandwidth-bound layers, it reduces execution-time overhead by 1.09–5.3× across diverse models, significantly improving efficiency over static ABFT approaches.

ABSTRACT

Neural networks (NNs) are increasingly employed in safety-critical domains and in environments prone to unreliability (e.g., soft errors), such as on spacecraft. Therefore, it is critical to impart fault tolerance to NN inference. Algorithm-based fault tolerance (ABFT) is emerging as an efficient approach for fault tolerance in NNs. We propose an adaptive approach to ABFT for NN inference that exploits untapped opportunities in emerging deployment scenarios. GPUs have high compute-to-memory-bandwidth ratios, while NN layers have a wide range of arithmetic intensities. This leaves some layers compute bound and others memory-bandwidth bound, but current approaches to ABFT do not consider these differences. We first investigate ABFT schemes best suited for each of these scenarios. We then propose intensity-guided ABFT, an adaptive, arithmetic-intensity-guided approach that selects the most efficient ABFT scheme for each NN layer. Intensity-guided ABFT reduces execution-time overhead by 1.09--5.3$ imes$ across many NNs compared to traditional approaches to ABFT.

Motivation & Objective

• To address the growing challenge of soft errors in safety-critical neural network deployments, especially in high-altitude and space environments.
• To overcome the inefficiency of existing ABFT approaches that assume all linear layers are compute-bound, ignoring resource bottlenecks in modern GPU-optimized inference.
• To exploit underutilized compute cycles in bandwidth-bound layers—common in low-arithmetic-intensity neural network layers—by introducing a thread-level ABFT scheme.
• To design an adaptive, per-layer ABFT strategy guided by arithmetic intensity to minimize execution-time overhead while ensuring fault detection.

Proposed method

• Analyzes the compute-to-memory-bandwidth ratio (CMR) of modern inference-optimized GPUs and identifies that many neural network layers are memory-bandwidth bound due to low arithmetic intensity.
• Proposes a thread-level ABFT scheme that performs checksum computations at the GPU thread level, eliminating inter-thread communication and avoiding additional memory accesses that would compete with the bandwidth-bound layer.
• Introduces intensity-guided ABFT, an adaptive framework that classifies each linear layer as compute-bound or bandwidth-bound based on its arithmetic intensity and selects the most efficient ABFT scheme accordingly.
• Employs a hybrid ABFT strategy: traditional ABFT for compute-bound layers and thread-level ABFT for bandwidth-bound layers, both integrated into the same inference pipeline.
• Uses the ratio of floating-point operations to bytes of memory traffic as a metric to determine arithmetic intensity and guide scheme selection.
• Evaluates the framework on multiple neural networks (e.g., ResNet, EfficientNet, MobileNet) across different GPU architectures, measuring execution-time overhead and fault detection coverage.

Experimental results

Research questions

• RQ1 Can underutilized compute resources in bandwidth-bound neural network layers on GPUs be leveraged for efficient redundant execution in fault tolerance?
• RQ2 How does the choice of ABFT scheme (traditional vs. thread-level) affect execution-time overhead across layers with varying arithmetic intensities?
• RQ3 Can an adaptive, arithmetic intensity-driven selection of ABFT schemes outperform static, one-size-fits-all ABFT approaches in terms of performance and fault coverage?
• RQ4 To what extent does thread-level ABFT reduce overhead compared to traditional ABFT in bandwidth-bound scenarios without introducing additional memory traffic?

Key findings

• Intensity-guided ABFT reduces execution-time overhead by 1.09–5.3× across multiple neural networks compared to traditional ABFT approaches.
• The proposed thread-level ABFT scheme achieves fault detection with zero additional memory traffic, making it ideal for bandwidth-bound layers.
• The framework successfully identifies and classifies layers as compute-bound or bandwidth-bound using arithmetic intensity as a guide, enabling optimal ABFT scheme selection per layer.
• Bandwidth-bound layers, which are common in modern, efficient neural network architectures, offer untapped opportunities for redundant computation with minimal performance cost.
• The adaptive selection strategy outperforms both static ABFT and replication-based approaches, especially in mixed workloads with varying layer characteristics.
• The approach is generalizable beyond neural networks, as it protects general matrix multiplications and can be applied to broader HPC workloads targeting GPU accelerators.