[Paper Review] Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy

Lucas Liebenwein, Cenk Baykal+3 more|arXiv (Cornell University)|Mar 4, 2021

Advanced Neural Network Applications63 references29 citations

TL;DR

The paper argues that pruning neural networks to maintain test accuracy can degrade robustness and generalization to distribution shifts; prune potential varies across tasks and data distributions, challenging the assumption of overparameterization.

ABSTRACT

Neural network pruning is a popular technique used to reduce the inference costs of modern, potentially overparameterized, networks. Starting from a pre-trained network, the process is as follows: remove redundant parameters, retrain, and repeat while maintaining the same test accuracy. The result is a model that is a fraction of the size of the original with comparable predictive performance (test accuracy). Here, we reassess and evaluate whether the use of test accuracy alone in the terminating condition is sufficient to ensure that the resulting model performs well across a wide spectrum of "harder" metrics such as generalization to out-of-distribution data and resilience to noise. Across evaluations on varying architectures and data sets, we find that pruned networks effectively approximate the unpruned model, however, the prune ratio at which pruned networks achieve commensurate performance varies significantly across tasks. These results call into question the extent of \emph{genuine} overparameterization in deep learning and raise concerns about the practicability of deploying pruned networks, specifically in the context of safety-critical systems, unless they are widely evaluated beyond test accuracy to reliably predict their performance. Our code is available at https://github.com/lucaslie/torchprune.

Motivation & Objective

• Motivate pruning as a means to reduce parameters while preserving performance, but question the sufficiency of test accuracy as the sole pruning criterion.
• Introduce functional distance metrics to compare pruned networks with their unpruned parents.
• Define prune potential as the maximal prune ratio that preserves nominal performance and study its variation across tasks and distributions.
• Investigate how pruning impacts robustness to distribution shifts, noise, and out-of-distribution data.
• Provide practical guidelines for pruning under distributional shifts and during retraining.

Proposed method

• Use iterative prune-retrain pipelines (unstructured and structured pruning) across datasets (CIFAR-10, ImageNet, PASCAL VOC) and architectures (ResNet, WRN, DenseNet, VGG).
• Compare pruning methods: weight thresholding (WT), SiPP, Filter Thresholding (FT), and Provable Filter Pruning (PFP).
• Formalize functional similarity with new distance metrics based on informative features and noise similarity.
• Define prune potential and excess loss to quantify robustness under distribution shifts (nominal vs. out-of-distribution data).
• Evaluate across norms of feature importance and layer-wise pruning budgets, and assess performance under corruptions (CIFAR10-C, ImageNet-C) and noise.
• Provide algorithmic description of PruneRetrain (Algorithm 1) and experimental setups.

Experimental results

Research questions

• RQ1 Do pruned networks remain functionally similar to their unpruned parents across tasks and prune ratios?
• RQ2 How does prune potential vary across tasks, architectures, and data distributions?
• RQ3 Do prune-enabled models maintain nominal test accuracy while degrading under out-of-distribution or noisy data?
• RQ4 Can pruning guidelines account for distribution shifts to avoid safety-critical failures?

Key findings

• Pruned models are functionally similar to the uncompressed parent, and can be distinguished from separately trained networks at various prune ratios.
• Prune potential varies significantly across tasks and data distributions, often lower on challenging inference tasks.
• Pruned networks tend to be more brittle under distribution shifts and noise, even when nominal accuracy matches the unpruned model.
• Performance under out-of-distribution data cannot be inferred from nominal test accuracy alone, raising safety concerns for deployment.
• Pruning guidelines suggest avoiding pruning when distribution shifts are likely, and using data augmentation or robust training to mitigate robustness losses.
• The study provides a framework to measure overparameterization via prune potential and recommends task-specific evaluation beyond test accuracy.