[Paper Review] To prune, or not to prune: exploring the efficacy of pruning for model compression

Michael Zhu, Suyog Gupta|arXiv (Cornell University)|Oct 5, 2017

Advanced Neural Network Applications661 citations

TL;DR

The paper compares large-sparse pruned models with small-dense ones across vision and language tasks, showing large-sparse models often outperform similarly-sized dense counterparts and introducing a simple gradual pruning method.

ABSTRACT

Model pruning seeks to induce sparsity in a deep neural network's various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model's dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across a broad range of neural network architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find large-sparse models to consistently outperform small-dense models and achieve up to 10x reduction in number of non-zero parameters with minimal loss in accuracy.

Motivation & Objective

• Motivate model compression for energy-efficient on-device inference.
• Evaluate two compression paths: large-sparse (pruned large model) vs small-dense (dense small model).
• Develop a simple gradual pruning technique easy to apply during training.

Proposed method

• Extend TensorFlow with binary masks per pruned layer to zero out small-magnitude weights during forward passes.
• Introduce a gradual sparsity schedule s_t that grows from s_i to s_f over n pruning steps, controlled by a cubic schedule: s_t = s_f + (s_i - s_f)(1 - (t - t_0)/(nΔt))^3.
• Update masks every Δt training steps to allow recovery from pruning-induced loss.
• Apply pruning across diverse architectures (InceptionV3, MobileNets, stacked LSTMs, seq2seq LSTMs, NMT).
• Compare large-sparse vs small-dense models with identical memory footprint across tasks.

Experimental results

Research questions

• RQ1 Can pruning a large model to achieve high sparsity outperform training a smaller dense model with the same memory footprint?
• RQ2 How does gradual pruning affect accuracy across vision and NLP architectures?
• RQ3 What are the practical hardware and storage considerations when using sparse vs dense models for on-device inference?
• RQ4 Is there an optimal sparsity level that maximizes accuracy for a given parameter budget?

Key findings

• Large-sparse models consistently outperform small-dense models at comparable memory footprints across tasks.
• In InceptionV3, 50% sparse achieves 13.6M NNZ with 78.0% top-1 and 94.2% top-5 accuracy, while 0% sparsity yields 27.1M NNZ, 78.1% top-1 and 94.3% top-5.
• At 87.5% sparsity, InceptionV3 drops to 3.3M NNZ with 74.6% top-1 and 92.5% top-5, a modest accuracy reduction given large compression.
• MobileNets pruned to 75% sparsity (1.09M NNZ) reach 67.7% top-1, outperforming the dense 0.75-width mobile net with the same NNZ budget; 90–95% sparse models maintain higher accuracy than equivalently-sized dense networks.
• On Penn Tree Bank, 90% sparse large models (6.6M NNZ) achieve perplexity 80.24, outperforming the dense medium model (19.8M NNZ) with perplexity 83.37; 85% sparse (3.0–3.0M NNZ) yields perplexities around 85.17–85.87, indicating an optimal compression range.
• In Google Neural Machine Translation, 90% sparse (23M NNZ) models achieve BLEU scores close to or better than much larger dense baselines, with 80% sparsity occasionally improving BLEU slightly; 90% sparse 1024-unit model (23M NNZ) is competitive with dense 512-unit model (81M parameters).
• Overall, large-sparse models exhibit a favorable trade-off, suggesting training a larger model and pruning yields better accuracy at a given size than pruning a smaller dense model.