[Paper Review] SPDY: Accurate Pruning with Speedup Guarantees

Elias Frantar, Dan Alistarh|arXiv (Cornell University)|Jan 31, 2022

Advanced Neural Network Applications4 citations

TL;DR

SPDY introduces a dynamic programming-based method to automatically determine optimal layer-wise sparsity profiles that guarantee a target inference speedup on specific hardware while minimizing accuracy loss. It combines an efficient dynamic programming solver with a local search procedure to learn layer-wise error scores from small calibration data, outperforming existing pruning methods in both one-shot and gradual pruning across vision and language models.

ABSTRACT

The recent focus on the efficiency of deep neural networks (DNNs) has led to significant work on model compression approaches, of which weight pruning is one of the most popular. At the same time, there is rapidly-growing computational support for efficiently executing the unstructured-sparse models obtained via pruning. Yet, most existing pruning methods minimize just the number of remaining weights, i.e. the size of the model, rather than optimizing for inference time. We address this gap by introducing SPDY, a new compression method which automatically determines layer-wise sparsity targets achieving a desired inference speedup on a given system, while minimizing accuracy loss. SPDY is composed of two new techniques: the first is an efficient dynamic programming algorithm for solving the speedup-constrained layer-wise compression problem assuming a set of given layer-wise sensitivity scores; the second is a local search procedure for determining accurate layer-wise sensitivity scores. Experiments across popular vision and language models show that SPDY guarantees speedups while recovering higher accuracy relative to existing strategies, both for one-shot and gradual pruning scenarios, and is compatible with most existing pruning approaches. We also extend our approach to the recently-proposed task of pruning with very little data, where we achieve the best known accuracy recovery when pruning to the GPU-supported 2:4 sparsity pattern.

Motivation & Objective

• Address the gap in existing pruning methods that optimize for model size rather than inference speed on specific hardware.
• Enable automatic, system-aware sparsity profile generation that guarantees a desired speedup while minimizing accuracy degradation.
• Overcome the limitations of heuristic or uniform sparsity assignment by learning layer-specific sensitivity scores from minimal calibration data.
• Ensure compatibility with both one-shot and gradual pruning, and extend to low-data and 2:4 sparsity settings.
• Provide a general framework applicable beyond unstructured pruning to other compression techniques like quantization or low-rank approximation.

Proposed method

• Formulate the layer-wise sparsity selection problem as a constrained optimization task, where the objective is to achieve a target speedup with minimal accuracy loss.
• Propose a dynamic programming algorithm with linear time complexity in the number of layers and sparsity choices per layer, enabling efficient and exact solution of the constrained compression problem.
• Introduce a local search procedure to automatically learn layer-wise error scores by evaluating the performance of reconstructed sparse models on a small calibration set.
• Use a modified version of AdaPrune for sparse model reconstruction, enabling accurate estimation of the impact of different sparsity profiles.
• Leverage global pruning behavior on calibration data to infer sensitivity scores that account for cross-layer effects, improving robustness over per-layer heuristics.
• Support arbitrary additive constraints (e.g., FLOPs, energy, parameter count), making the framework extensible beyond speedup.

Experimental results

Research questions

• RQ1 Can a general method be developed to automatically determine layer-wise sparsity levels that guarantee a specific inference speedup on a given hardware platform?
• RQ2 How can layer-wise error scores be learned in a way that captures global network behavior and improves accuracy over standard heuristics like weight magnitude?
• RQ3 Does the proposed method outperform existing pruning strategies in terms of accuracy and speedup consistency across diverse models and pruning regimes?
• RQ4 Can the method be extended to low-data or post-training 2:4 sparsity settings with strong performance?
• RQ5 To what extent do the learned sparsity profiles generalize across different training regimes, such as one-shot vs. gradual pruning?

Key findings

• SPDY achieves higher accuracy than state-of-the-art baselines across all tested models, including ResNet50, YOLOv5m, and BERT, especially at higher speedups.
• For YOLOv5m, SPDY achieves a mAP@0.5 of 61.50 at 2.00× speedup, outperforming the next best method by over 2.5 points.
• On BERT SQuAD, SPDY maintains 87.14 F1 at 4.00× speedup, outperforming the next best method by 1.4 points in F1-drop reduction.
• The method reduces overall model sparsity compared to uniform or GMP pruning while achieving the same speedup, which likely contributes to higher accuracy after fine-tuning.
• Iterative application of SPDY—re-running from an intermediate pruned model—leads to further accuracy gains, suggesting potential for progressive refinement.
• SPDY profiles generalize well: when applied to AC/DC-trained models, they still yield superior performance, indicating architecture-aware transferability akin to lottery ticket principles.