[Paper Review] Network Pruning That Matters: A Case Study on Retraining Variants

Duong Hoang Le, Binh‐Son Hua|arXiv (Cornell University)|May 7, 2021

Advanced Neural Network Applications37 references17 citations

TL;DR

This paper investigates the critical role of learning rate scheduling in retraining pruned neural networks, demonstrating that large learning rates—particularly via learning rate restarting—significantly outperform traditional fine-tuning. Surprisingly, randomly pruned networks retrained with large learning rates can surpass methodically pruned networks, revealing that retraining hyperparameters matter more than pruning methodology itself.

ABSTRACT

Network pruning is an effective method to reduce the computational expense of over-parameterized neural networks for deployment on low-resource systems. Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the traditional fine-tuning technique in recovering the lost accuracy (Renda et al., 2020), but so far it is unclear what accounts for such performance. In this work, we conduct extensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding. We find that the reason behind the success of learning rate rewinding is the usage of a large learning rate. Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e.g., the 1-cycle learning rate schedule (Smith et al., 2019). By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (fine-tuned with the conventional approach). Our results emphasize the cruciality of the learning rate schedule in pruned network retraining - a detail often overlooked by practitioners during the implementation of network pruning. One-sentence Summary: We study the effective of different retraining mechanisms while doing pruning

Motivation & Objective

• To investigate why learning rate rewinding (LRW) outperforms traditional fine-tuning in retraining pruned networks.
• To determine whether the success of LRW is due to the specific schedule or the use of large learning rates.
• To evaluate whether simple retraining with large learning rates can outperform complex pruning algorithms.
• To advocate for standardized retraining configurations in fair comparisons of pruning algorithms.
• To highlight the overlooked importance of learning rate scheduling in network pruning pipelines.

Proposed method

• Extensive experiments comparing various retraining strategies, including fine-tuning, learning rate rewinding (LRW), and learning rate restarting (CLR).
• Application of the 1-cycle learning rate schedule (CLR) to retrain pruned models, particularly focusing on large initial learning rates.
• Random pruning of filters using uniformly random importance scores, followed by retraining with large learning rates to assess performance.
• Use of standard training recipes with consistent hyperparameters across methods to isolate the effect of learning rate scheduling.
• Comparison of performance across multiple datasets (CIFAR-10, ImageNet) and models (ResNet, DenseNet, VGG, ResNet-50) under varying compression ratios.
• Ablation studies to isolate the impact of learning rate magnitude and schedule on final accuracy after pruning.

Experimental results

Research questions

• RQ1 Why does learning rate rewinding (LRW) outperform traditional fine-tuning in retraining pruned networks?
• RQ2 Is the success of LRW due to the specific schedule or the use of large learning rates?
• RQ3 Can a simple retraining strategy with large learning rates outperform complex pruning algorithms?
• RQ4 Does the choice of retraining hyperparameters affect the fairness of comparisons between pruning methods?
• RQ5 Can randomly pruned networks achieve performance comparable to or better than methodically pruned networks when retrained with optimal learning rates?

Key findings

• Learning rate rewinding (LRW) outperforms fine-tuning primarily due to the use of a large learning rate, not the specific schedule coupling with original training.
• Retraining with a large learning rate via the 1-cycle schedule (CLR) consistently improves performance across all models and datasets, regardless of pruning method or compression ratio.
• Randomly pruned networks retrained with CLR achieved higher accuracy than methodically pruned networks using the same pruning criteria, demonstrating that retraining hyperparameters can outweigh pruning strategy.
• On CIFAR-10, random pruning with CLR achieved 93.40±0.16% top-1 accuracy on ResNet-110, surpassing HRankPlus’s 93.37±0.04% and MWP’s 93.23%.
• On ImageNet, random pruning with CLR achieved 75.54% top-1 accuracy on ResNet-50 at 81% compression, exceeding Taylor Pruning’s 75.48% and PFEC’s 73.47% at similar compression levels.
• The study reveals that retraining with large learning rates can make simple norm-based pruning competitive with complex pruning algorithms, challenging the assumption that sophisticated pruning metrics are essential.