[Paper Review] Accelerating Sparse Deep Neural Networks

Asit Mishra, Jorge Albericio Latorre+6 more|arXiv (Cornell University)|Apr 16, 2021

Advanced Neural Network Applications52 references14 citations

TL;DR

This paper introduces a 2:4 structured sparsity pattern that enables 2× higher math throughput on NVIDIA's Sparse Tensor Cores in the Ampere GPU architecture. By pruning dense models to this structured pattern and retraining once with fixed hyperparameters, the method maintains model accuracy across diverse architectures while enabling hardware-accelerated inference with no hyperparameter tuning.

ABSTRACT

As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.

Motivation & Objective

• Address the challenge of achieving both high model accuracy and hardware acceleration in sparse deep neural networks.
• Overcome the limitations of unstructured sparsity, which fails to leverage modern GPU matrix-math pipelines.
• Develop a practical, hardware-aware sparsity pattern that enables significant speedups without accuracy loss.
• Provide a universal training workflow that maintains accuracy after pruning by retraining with fixed hyperparameters.
• Enable efficient deployment of sparse models on modern GPU hardware through native support in Tensor Cores.

Proposed method

• Propose a 2:4 structured sparsity pattern where every group of four consecutive values contains at least two zeros, enabling efficient hardware compression and computation.
• Design a compressed storage format to efficiently represent 2:4 sparse tensors in memory, reducing bandwidth and storage requirements.
• Introduce Sparse Tensor Cores in the NVIDIA Ampere GPU architecture that achieve 2× higher math throughput for GEMM operations when the first operand is 2:4 sparse.
• Develop a two-stage training workflow: first, prune a fully trained dense model using magnitude-based criteria; second, retrain the pruned model with the same hyperparameters as the original training.
• Enforce the 2:4 sparsity pattern as a fixed mask during retraining, ensuring hardware compatibility without requiring dynamic sparsity or hyperparameter search.
• Apply the workflow to a wide range of models including ResNet, EfficientNet, BERT, and vision transformers, demonstrating consistent accuracy maintenance.

Experimental results

Research questions

• RQ1 Can a structured sparsity pattern be designed to enable both high hardware utilization and high model accuracy in deep neural networks?
• RQ2 Can a universal training workflow be developed that maintains model accuracy after pruning without requiring hyperparameter tuning or re-optimization?
• RQ3 To what extent can hardware acceleration via Sparse Tensor Cores achieve performance gains on real-world models while preserving accuracy?
• RQ4 Does enforcing a 2:4 sparsity pattern allow for significant speedups on modern GPU architectures without sacrificing model performance?
• RQ5 Can the same training workflow be applied universally across diverse architectures and tasks without modification?

Key findings

• The 2:4 structured sparsity pattern enables 2× higher math throughput on NVIDIA's Sparse Tensor Cores for GEMM operations, directly accelerating inference in DNNs.
• The proposed retraining workflow maintains model accuracy across a wide range of architectures—including ResNet-50, EfficientNet-B0, BERT-Large, and vision transformers—without changing any hyperparameters.
• On ResNet-50, the method achieved a 2× speedup with only 1.5% accuracy loss using standard channel pruning, but with the proposed workflow, accuracy was fully recovered.
• For BERT-Large, the pruned and retrained model matched the accuracy of the dense model even after INT8 quantization, demonstrating robustness to mixed-precision inference.
• The method was empirically validated on multiple tasks including image classification, object detection, and natural language understanding, showing consistent performance and accuracy.
• Preliminary results suggest that activation pruning with the same 2:4 pattern is feasible without accuracy loss, indicating potential for broader application beyond weights.