[Paper Review] Layer-adaptive sparsity for the Magnitude-based Pruning

Jaeho Lee, Sejun Park+3 more|arXiv (Cornell University)|Oct 15, 2020

Advanced Neural Network Applications45 references69 citations

TL;DR

LAMP introduces a layer-adaptive magnitude-based pruning score that globally prunes weights by a layer-aware distortion measure, achieving better sparsity-accuracy tradeoffs without hyperparameter tuning across diverse architectures and datasets.

ABSTRACT

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on "how to choose," the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity

Motivation & Objective

• Motivate a go-to layerwise sparsity scheme for magnitude-based pruning without manual tuning.
• Develop a model-level distortion minimization perspective to derive a practical pruning score.
• Propose LAMP, a hyperparameter-free score that preserves model output integrity under pruning.
• Validate LAMP across multiple architectures and datasets with and without weight rewinding.
• Analyze how LAMP-derived sparsity patterns compare to handcrafted heuristics and prior baselines.

Proposed method

• Define the LAMP score as score(u;W) = (W[u])^2 / sum_{v>=u} (W[v])^2 for each weight when weights are ordered by magnitude within a layer.
• Apply a global pruning by selecting the smallest LAMP scores to meet a global sparsity constraint, which is equivalent to magnitude-based pruning with automatically chosen layerwise sparsity.
• Justify LAMP from an output distortion perspective by showing layerwise MP solves a relaxed Frobenius distortion minimization, leading to a greedy, distortion-minimizing pruning process.
• Demonstrate that LAMP can be computed with minimal overhead, requiring only a layer-wise cumulative sum of squared weights for the denominator.
• Show that LAMP is hyperparameter-free and does not rely on model-specific knowledge, yet approximates model-level distortion well.
• Evaluate LAMP with multiple CNN architectures (VGG-16, ResNet-18/34, DenseNet-121, EfficientNet-B0) across CIFAR-10/100, SVHN, and Restricted ImageNet.

Experimental results

Research questions

• RQ1 Can a layer-adaptive, hyperparameter-free pruning score outperform existing layerwise sparsity selection schemes for magnitude-based pruning?
• RQ2 Does the LAMP score effectively approximate model-level output distortion under pruning across diverse architectures and datasets?
• RQ3 How does LAMP interact with weight rewinding and one-shot pruning compared to other baselines?
• RQ4 What sparsity patterns does LAMP induce across layers, and how do they compare to handcrafted heuristics like keeping the first layer dense or last layer sparsity constraints?

Key findings

• LAMP consistently outperforms baseline layerwise sparsity schemes in sparsity-accuracy tradeoffs across VGG-16, ResNet-18/34, DenseNet-121, and EfficientNet-B0 on CIFAR-10.
• On CIFAR-10, LAMP achieves 88.1% test accuracy with only 1.44% of weights surviving, outperforming Erdős-Rényi kernel which achieves 77.8%.
• LAMP also outperforms baselines on SVHN, CIFAR-100, and Restricted ImageNet in the reported experiments, with Erdős-Rényi kernel remaining competitive in some cases.
• LAMP performs reliably well under one-shot pruning and weight rewinding, whereas Erdős-Rényi can be sensitive to rewinding.
• In SNIP settings, LAMP attains performance comparable to Global SNIP, suggesting robustness beyond standard MP scenarios.
• Layerwise sparsity induced by LAMP tends to keep first and last layers relatively denser, and promotes a more uniform distribution of nonzero weights across layers at high sparsity.