[Paper Review] Rethinking the Value of Network Pruning

Zhuang Liu, Mingjie Sun+3 more|arXiv (Cornell University)|Oct 11, 2018

Anomaly Detection Techniques and Applications742 citations

TL;DR

The paper shows that for structured pruning, training the pruned model from scratch often matches or surpasses fine-tuning with inherited weights, and the pruned architecture itself is the key driver of efficiency, suggesting pruning can serve as architecture search.

ABSTRACT

Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.

Motivation & Objective

• Question the necessity of training a large over-parameterized model before pruning.
• Evaluate whether fine-tuning pruned models with inherited weights outperforms training pruned models from scratch.
• Distinguish effects of predefined vs automatic (architecture-discovering) pruning targets.
• Assess whether pruning serves mainly as architecture search rather than weight selection.
• Compare structured pruning with unstructured pruning and relate findings to the Lottery Ticket Hypothesis.

Proposed method

• Classify pruning into predefined target architectures and automatically discovered target architectures.
• Train pruned models from scratch (Scratch-E, Scratch-B) vs fine-tune from inherited weights.
• Apply multiple pruning methods (L1-norm filter pruning, ThiNet, Regression-based reconstruction, Network Slimming, Sparse Structure Selection) and an unstructured magnitude-based prune.
• Evaluate on CIFAR-10, CIFAR-100, and ImageNet across VGG, ResNet, and DenseNet variants.
• Analyze parameter efficiency and sparsity patterns of pruned architectures.
• Compare with Lottery Ticket Hypothesis and discuss implications for architecture search.

Experimental results

Research questions

• RQ1 Does fine-tuning a pruned model with inherited weights outperform training the same pruned architecture from scratch across predefined and automatic pruning targets?
• RQ2 To what extent does the pruned architecture, rather than preserved weights, determine the final efficiency and accuracy?
• RQ3 Can pruning act as an effective architecture search method, yielding parameter-efficient architectures without large-model pretraining?
• RQ4 How do structured and unstructured pruning compare in the ability to train pruned models from scratch on large-scale datasets like ImageNet?

Key findings

• For predefined structured pruning, scratch-trained models reach or exceed the accuracy of fine-tuned counterparts, with Scratch-B often better than Scratch-E and sometimes outperforming fine-tuning on ImageNet.
• For automatic structured pruning, scratch-trained pruned models generally match or beat fine-tuned models, with Scratch-B frequently superior.
• Unstructured pruning on ImageNet shows scratch training performing worse than fine-tuning, highlighting differences from structured pruning.
• Pruned architectures obtained via automatic pruning methods are more parameter efficient than uniformly pruned architectures, indicating architecture search value.
• Guided/pruned architectures can transfer design patterns to other models/datasets, suggesting practical design principles beyond the specific pruned models.