[Paper Review] A Deeper Look at the Layerwise Sparsity of Magnitude-based Pruning.

Jaeho Lee, Sejun Park+3 more|arXiv (Cornell University)|Oct 15, 2020

Advanced Neural Network Applications41 references4 citations

TL;DR

This paper proposes Layer-Adaptive Magnitude-Based Pruning (LAMP), a novel importance score for global magnitude-based pruning that adaptively determines layerwise sparsity by rescaling weight magnitudes with model-level ℓ₂ distortion, eliminating hyperparameter tuning. LAMP achieves state-of-the-art performance across diverse models and datasets, outperforming existing methods—including connectivity-oriented sparsity—especially in weight-rewinding settings where baselines degrade.

ABSTRACT

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on how to choose, the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under diverse datasets and models, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case.

Motivation & Objective

• To address the lack of consensus in selecting optimal layerwise sparsity for magnitude-based pruning.
• To eliminate reliance on handcrafted heuristics or extensive hyperparameter searches in pruning strategy design.
• To develop a principled, computationally efficient importance score that adapts sparsity per layer based on global model impact.
• To improve the performance of global magnitude pruning under diverse training regimes, including weight-rewinding.
• To demonstrate that connectivity-based sparsity, while strong in standard settings, underperforms in dynamic training setups like weight rewinding.

Proposed method

• LAMP introduces a rescaled weight magnitude score that accounts for the ℓ₂ distortion caused by pruning across the entire model.
• The importance score is derived from the magnitude of weights, adjusted by a layer-specific normalization that reflects the impact of pruning on model performance.
• The method does not require additional forward passes or gradient computation, making it computationally lightweight.
• Layerwise sparsity is determined directly from the LAMP score without hyperparameter tuning or validation.
• The approach is applied globally across all layers, enabling end-to-end pruning with consistent performance.
• The method is evaluated under standard and weight-rewinding training protocols to assess robustness.

Experimental results

Research questions

• RQ1 How does LAMP compare to existing layerwise sparsity selection methods in terms of accuracy and sparsity trade-offs across different models and datasets?
• RQ2 Does LAMP maintain superior performance under weight-rewinding, where model dynamics change significantly?
• RQ3 Why does connectivity-oriented layerwise sparsity fail in weight-rewinding scenarios, while LAMP remains robust?
• RQ4 Can a simple, hyperparameter-free importance score outperform complex, tuned baselines in global pruning?
• RQ5 What is the role of model-level ℓ₂ distortion in guiding effective layerwise sparsity allocation?

Key findings

• LAMP consistently outperforms popular layerwise sparsity selection methods across multiple datasets and model architectures.
• LAMP achieves state-of-the-art accuracy-sparsity trade-offs without requiring any hyperparameter tuning or additional computation.
• In weight-rewinding setups, LAMP maintains strong performance, while connectivity-oriented sparsity methods degrade significantly.
• The performance gap between LAMP and baselines widens in dynamic training settings, highlighting LAMP's robustness.
• The ℓ₂ distortion-aware rescaling in LAMP effectively captures the true impact of pruning per layer, leading to better sparsity allocation.
• LAMP's success demonstrates that model-level global impact can guide layerwise pruning more effectively than connectivity-based heuristics.