[Paper Review] A Unified DNN Weight Compression Framework Using Reweighted Optimization Methods

Tianyun Zhang, Xiaolong Ma+8 more|arXiv (Cornell University)|Apr 12, 2020

Advanced Data Compression Techniques34 references4 citations

TL;DR

This paper proposes a unified DNN weight pruning framework using reweighted optimization to achieve high-accuracy, high-rate structured and non-structured sparsity with automatic hyperparameter tuning. By replacing ADMM's hard $ε$-norm constraints with dynamically updated reweighted $¹$ regularization, the method accelerates convergence, reduces training time from 150 to 85 epochs, and achieves 4.2× structured pruning on ResNet-18 with 88.5% top-5 accuracy—surpassing state-of-the-art ADMM methods.

ABSTRACT

To address the large model size and intensive computation requirement of deep neural networks (DNNs), weight pruning techniques have been proposed and generally fall into two categories, i.e., static regularization-based pruning and dynamic regularization-based pruning. However, the former method currently suffers either complex workloads or accuracy degradation, while the latter one takes a long time to tune the parameters to achieve the desired pruning rate without accuracy loss. In this paper, we propose a unified DNN weight pruning framework with dynamically updated regularization terms bounded by the designated constraint, which can generate both non-structured sparsity and different kinds of structured sparsity. We also extend our method to an integrated framework for the combination of different DNN compression tasks.

Motivation & Objective

• To address the limitations of static and dynamic regularization-based DNN pruning, including accuracy degradation and long convergence times.
• To unify non-structured and structured sparsity generation under a single optimization framework with reduced hyperparameter sensitivity.
• To accelerate convergence and reduce training time by replacing non-convex $ε$-norm constraints with reweighted $¹$ regularization.
• To enable automatic per-layer pruning rate determination through dynamic regularization, eliminating manual tuning.
• To extend the framework to integrated compression tasks, such as combining pruning and quantization.

Proposed method

• The method uses reweighted $¹$ regularization as a proxy for the intractable $ε$-norm sparsity constraint, enabling efficient optimization.
• It dynamically updates the reweighting coefficients based on current weight magnitudes to progressively identify and prune less important weights.
• Weights below a threshold are masked from gradient updates and permanently removed, while remaining weights are fine-tuned to recover accuracy.
• The framework supports both non-structured sparsity and various structured sparsity patterns via group lasso with reweighted regularization.
• For multi-task compression, the method combines ADMM-based hard constraints for one task (e.g., pattern pruning) with reweighted regularization for another (e.g., weight quantization).
• The approach avoids the strong non-convexity of $ε$-norm constraints, enabling faster convergence and fewer hyperparameters than ADMM.

Experimental results

Research questions

• RQ1 Can reweighted $¹$ regularization outperform ADMM-based hard constraints in terms of pruning rate and convergence speed?
• RQ2 Can a unified framework generate both non-structured and structured sparsity with minimal hyperparameter tuning?
• RQ3 Does combining reweighted regularization with ADMM for multi-task compression yield better total compression rates than using ADMM for both tasks?
• RQ4 Can the method automatically determine per-layer pruning rates without manual configuration?
• RQ5 How does the method compare to magnitude-based and $¹$-based pruning in accuracy and compression efficiency?

Key findings

• The proposed method achieves 4.2× structured pruning on ResNet-18 for ImageNet with 88.5% top-5 accuracy, outperforming state-of-the-art ADMM methods (3.0×, 87.9% accuracy).
• Training time is reduced from 150 epochs (ADMM) to 85 epochs using a single penalty parameter, eliminating per-layer hyperparameter tuning.
• On LeNet-5 for MNIST, the method achieves 1014× total compression rate with 0.2% accuracy loss, surpassing ADMM-NN's 623× rate.
• On AlexNet for ImageNet, the method achieves 115× total compression rate with 0.2% accuracy loss, exceeding ADMM-NN's 99× rate.
• For combined pattern and kernel-wise pruning on VGG-16, the method achieves 34.9× pruning rate on CIFAR-10 (1.76× higher than PCONV) and 13.1× on ImageNet (1.87× higher than PCONV).
• The reweighted regularization framework enables automatic per-layer pruning rates and reduces dependency on heuristic hyperparameter selection.