[Paper Review] Pruning Neural Networks at Initialization: Why are We Missing the Mark?

Jonathan Frankle, Gintare Karolina Dziugaite+2 more|arXiv (Cornell University)|Sep 18, 2020

Advanced Neural Network Applications35 references48 citations

TL;DR

The paper evaluates SNIP, GraSP, SynFlow, and magnitude pruning at initialization, showing they beat random pruning but generally underperform magnitude pruning after training; pruning decisions are largely captured by per-layer pruning proportions rather than individual weights.

ABSTRACT

Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.

Motivation & Objective

• Assess how well pruning-at-initialization methods (SNIP, GraSP, SynFlow) perform compared with magnitude pruning and random baselines.
• Understand whether the pruning decisions are per-weight or per-layer in initialization-based pruning.
• Identify ablations that reveal the information these methods actually use.
• Determine whether pruning at initialization can match pruning after training across common architectures and sparsities.

Proposed method

• Define pruning to sparsity s via per-weight scores and removal routines.
• Compare one-shot pruning at initialization across SNIP, GraSP, SynFlow, magnitude pruning, and baselines.
• Perform ablations: weight shuffle, reinitialization, and inversion to test sensitivity.
• Evaluate on CNNs/ResNets (CIFAR-10, TinyImageNet, ImageNet) at matching sparsities where post-training magnitude pruning matches full accuracy.
• Extend analysis to pruning after initialization by pruning later in training and retraining.

Experimental results

Research questions

• RQ1 How do SNIP, GraSP, SynFlow, and magnitude pruning at initialization compare to random pruning and magnitude pruning after training across standard networks?
• RQ2 Are the pruning decisions at initialization sensitive to per-weight choices or mainly to per-layer sparsity proportions?
• RQ3 Do ablations (shuffle, reinitialization, inversion) reveal fundamental differences between pruning-at-initialization methods and pruning-after-training?
• RQ4 Can pruning-at-initialization achieve comparable accuracy to pruning-after-training by pruning later in training or with different signals?

Key findings

• All pruning-at-initialization methods outperform random pruning at matching sparsities, with no single method universally best across networks.
• Magnitude pruning after training often achieves higher accuracy and sparser matching sparsities than initialization-time pruning.
• Pruning decisions in SNIP, GraSP, SynFlow, and magnitude pruning at initialization are largely captured by layerwise pruning proportions; shuffling within layers preserves performance.
• Reinitializing or shuffling unpruned weights does not hurt initialization-time pruning performance, unlike pruning-after-training methods.
• SynFlow can exhibit neuron-collapse behavior under certain extreme sparsities, pruning whole neurons more aggressively.
• When pruning after initialization, these methods’ accuracy improves less rapidly than magnitude pruning after training, suggesting intrinsic limitations of pruning at initialization; pruning later in training (LTR baseline) can outperform early methods.
• Pruning later in training with these methods (SNIP, SynFlow, magnitude) improves accuracy but still lags behind LTR at the most extreme sparsities, indicating broader challenges with pruning at initialization.