[Paper Review] DropPruning for Model Compression.

Haipeng Jia, Xueshuang Xiang+6 more|arXiv (Cornell University)|Dec 5, 2018

Advanced Neural Network Applications4 citations

TL;DR

This paper introduces Drop Pruning, a novel iterative model compression technique that uses stochastic 'drop out' (removing unimportant weights) and 'drop in' (recovering pruned weights) strategies to compress deep neural networks without accuracy loss. It achieves state-of-the-art compression ratios—11.1× for VGG-16 on CIFAR-10 and 14.3× for LeNet-5 on MNIST—by dynamically managing sparsity while reducing overfitting through a dropout-inspired mechanism.

ABSTRACT

Deep neural networks (DNNs) have dramatically achieved great success on a variety of challenging tasks. However, most of the successful DNNs are structurally so complex, leading to much storage requirement and floating-point operation. This paper proposes a novel technique, named Drop Pruning, to compress the DNNs by pruning the weights from a dense high-accuracy baseline model without accuracy loss. Drop Pruning also falls into the standard iterative prune-retrain procedure, where a \emph{drop} strategy exists at each pruning step: \emph{drop out}, stochastic deleting some unimportant weights and \emph{drop in}, stochastic recovering some pruned weights. \emph{Drop out} and \emph{drop in} are supposed to handle the two drawbacks of the traditional pruning methods: local importance judgment and irretrievable pruning process, respectively. The suitable choosing of \emph{drop} probabilities can decrease the model size during pruning process and lead it to flow to the target sparsity. Drop Pruning also has some similar spirits with dropout, a stochastic algorithm in Integer Optimization and the Dense-Sparse-Dense training technique. Drop Pruning can significantly reducing overfitting while compressing the model. Experimental results demonstrates that Drop Pruning can achieve the state-of-the-art performance on many benchmark pruning tasks, about ${11.1 imes}$ compression of VGG-16 on CIFAR10 and ${14.3 imes}$ compression of LeNet-5 on MNIST without accuracy loss, which may provide some new insights into the aspect of model compression.

Motivation & Objective

• To address the limitations of traditional weight pruning methods, such as local importance estimation and irreversible pruning decisions.
• To reduce model overfitting during the pruning and retraining process.
• To enable flexible and accurate control over model sparsity during compression.
• To achieve high compression ratios while maintaining the original model's accuracy.
• To explore the synergy between stochastic pruning and training techniques like dropout and Dense-Sparse-Dense training.

Proposed method

• Introduces a novel iterative prune-retrain framework that alternates between 'drop out' and 'drop in' phases to dynamically manage weight importance.
• Applies stochastic weight removal ('drop out') based on importance scores to eliminate unimportant connections during pruning.
• Uses stochastic recovery ('drop in') to reintroduce previously pruned weights, mitigating the risk of irreversible pruning.
• Employs adaptive drop probabilities to guide the model toward a target sparsity level while maintaining accuracy.
• Leverages principles from dropout and Dense-Sparse-Dense training to improve generalization and stability during compression.
• Maintains a high-accuracy baseline model throughout the process, ensuring no accuracy degradation after pruning.

Experimental results

Research questions

• RQ1 Can a stochastic pruning strategy that includes both removal and recovery of weights improve model compression without accuracy loss?
• RQ2 How does the dynamic interplay between 'drop out' and 'drop in' reduce overfitting during iterative pruning?
• RQ3 To what extent can Drop Pruning achieve higher compression ratios than existing methods on standard benchmarks?
• RQ4 How does the choice of drop probabilities influence the convergence and sparsity of the compressed model?
• RQ5 Does Drop Pruning inherit the generalization benefits of dropout and Dense-Sparse-Dense training in compressed models?

Key findings

• Drop Pruning achieves 11.1× compression of VGG-16 on CIFAR-10 without any accuracy loss.
• The method achieves 14.3× compression of LeNet-5 on MNIST while preserving the original model's accuracy.
• The stochastic 'drop in' mechanism effectively reverses irreversible pruning decisions, improving robustness.
• Drop Pruning significantly reduces overfitting during the iterative pruning and retraining process.
• The approach demonstrates state-of-the-art performance across multiple benchmark pruning tasks.
• The technique successfully integrates principles from dropout and Dense-Sparse-Dense training to enhance model generalization.