[Paper Review] Neural Pruning via Growing Regularization

Huan Wang, Can Qin+2 more|arXiv (Cornell University)|Dec 16, 2020

Domain Adaptation and Few-Shot Learning63 references4 citations

TL;DR

This paper proposes a growing regularization approach that gradually increases L2 penalty strength to improve neural network pruning, addressing both pruning schedule and weight importance scoring. By leveraging Hessian-induced magnitude separation without explicit Hessian computation, the method achieves state-of-the-art accuracy across structured and unstructured pruning on CIFAR-10/100 and ImageNet.

ABSTRACT

Regularization has long been utilized to learn sparsity in deep neural network pruning. However, its role is mainly explored in the small penalty strength regime. In this work, we extend its application to a new scenario where the regularization grows large gradually to tackle two central problems of pruning: pruning schedule and weight importance scoring. (1) The former topic is newly brought up in this work, which we find critical to the pruning performance while receives little research attention. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains compared with its one-shot counterpart, even when the same weights are removed. (2) The growing penalty scheme also brings us an approach to exploit the Hessian information for more accurate pruning without knowing their specific values, thus not bothered by the common Hessian approximation problems. Empirically, the proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning. Their effectiveness is demonstrated with modern deep neural networks on the CIFAR and ImageNet datasets, achieving competitive results compared to many state-of-the-art algorithms. Our code and trained models are publicly available at https://github.com/mingsuntse/regularization-pruning.

Motivation & Objective

• Address the underexplored problem of pruning schedule in neural network pruning, which significantly impacts performance but receives little research attention.
• Develop a method to exploit Hessian information for accurate weight importance scoring without requiring explicit Hessian computation or approximation.
• Enable effective unstructured and structured pruning through a scalable, simple-to-implement regularization scheme that enhances model expressivity during pruning.
• Demonstrate that magnitude-based pruning can achieve Hessian-based pruning performance when combined with growing regularization, bridging two major pruning paradigms.

Proposed method

• Propose a growing L2 regularization scheme where the penalty factor increases gradually over training, rather than being fixed at a small value.
• Use the increasing penalty to push unimportant weights toward zero, enabling accurate pruning using simple magnitude-based criteria.
• Leverage the local quadratic structure (Hessian) of weights implicitly: weights with higher curvature are penalized more strongly, leading to better separation in magnitude.
• Design two variants: GReg-1 for structured pruning (filter-level) and GReg-2 for unstructured pruning, both using increasing L2 penalties.
• Apply the method to both small-scale (CIFAR) and large-scale (ImageNet) datasets, with standard training protocols for fair comparison.
• Avoid Hessian approximation by relying on the natural dynamics of growing regularization to reflect curvature differences across weights.

Experimental results

Research questions

• RQ1 Can a growing regularization schedule significantly improve pruning performance compared to one-shot pruning with fixed regularization?
• RQ2 Can growing regularization implicitly exploit Hessian information for weight importance scoring without requiring explicit Hessian computation or approximation?
• RQ3 Does the proposed method achieve competitive or superior accuracy compared to state-of-the-art structured and unstructured pruning methods on large-scale benchmarks?
• RQ4 Is the pruning schedule as critical as the weight importance criterion in determining final model performance?
• RQ5 Can magnitude-based pruning achieve Hessian-based pruning accuracy through a growing regularization framework?

Key findings

• On ResNet50 with ~2.5× speedup on ImageNet, GReg-2 achieves 74.93% top-1 accuracy, outperforming Taylor-FO (71.69%) by 3.24 percentage points and matching the best-performing methods.
• At ~3× speedup on ImageNet, GReg-2 achieves 73.90% top-1 accuracy, outperforming Taylor-FO by 2.23 percentage points, demonstrating superior performance at higher sparsity.
• On CIFAR-100 with 2.5× speedup, GReg-2 achieves 74.85% top-1 accuracy, outperforming the baseline by 1.28 percentage points and matching or exceeding state-of-the-art methods.
• In unstructured pruning on ImageNet, GReg-2 achieves 75.45% top-1 accuracy at 82.7% sparsity, outperforming WoodFisher (75.20%) in accuracy and matching it in drop, despite not using Hessian approximation.
• GReg-1, a simpler variant, achieves competitive results with 75.45% top-1 accuracy at 82.7% sparsity, showing that improved pruning schedule alone can yield strong performance.
• The method is scalable and effective across both structured and unstructured pruning, with no need for complex Hessian estimation or iterative retraining.