[Paper Review] Accelerate CNN via Recursive Bayesian Pruning

Yuefu Zhou, Ya Zhang+2 more|arXiv (Cornell University)|Dec 2, 2018

Advanced Neural Network Applications44 references4 citations

TL;DR

This paper proposes Recursive Bayesian Pruning (RBP), a layer-wise Bayesian channel pruning method that models inter-layer dependency by treating dropout noise across layers as a Markov chain, enabling efficient, data-driven pruning without additional overhead. RBP achieves up to 5.0× FLOPs reduction with minimal accuracy drop on ImageNet for VGG16 and 2.2× for ResNet50.

ABSTRACT

Channel Pruning, widely used for accelerating Convolutional Neural Networks, is an NP-hard problem due to the inter-layer dependency of channel redundancy. Existing methods generally ignored the above dependency for computation simplicity. To solve the problem, under the Bayesian framework, we here propose a layer-wise Recursive Bayesian Pruning method (RBP). A new dropout-based measurement of redundancy, which facilitate the computation of posterior assuming inter-layer dependency, is introduced. Specifically, we model the noise across layers as a Markov chain and target its posterior to reflect the inter-layer dependency. Considering the closed form solution for posterior is intractable, we derive a sparsity-inducing Dirac-like prior which regularizes the distribution of the designed noise to automatically approximate the posterior. Compared with the existing methods, no additional overhead is required when the inter-layer dependency assumed. The redundant channels can be simply identified by tiny dropout noise and directly pruned layer by layer. Experiments on popular CNN architectures have shown that the proposed method outperforms several state-of-the-arts. Particularly, we achieve up to $\bf{5.0 imes}$ and $\bf{2.2 imes}$ FLOPs reduction with little accuracy loss on the large scale dataset ILSVRC2012 for VGG16 and ResNet50, respectively.

Motivation & Objective

• Address the challenge of channel pruning in CNNs, where inter-layer dependencies complicate redundancy identification.
• Overcome the limitation of existing Bayesian pruning methods that assume independent channels across layers.
• Develop a pruning framework that captures inter-layer dependency without introducing computational overhead.
• Enable efficient, layer-wise pruning guided by Bayesian inference with sparsity-inducing priors.
• Balance model acceleration with data fitness by preserving model performance through fast fine-tuning after pruning.

Proposed method

• Model dropout noise across layers as a Markov chain to encode inter-layer dependency in the Bayesian framework.
• Introduce a sparsity-inducing Dirac-like prior to regularize the distribution of dropout noise and approximate the intractable posterior.
• Use variational inference with the Dirac-like prior to identify redundant channels via near-zero dropout rates.
• Perform pruning layer by layer based on the inferred dropout rates, with no additional inference cost.
• Leverage reparameterization tricks to improve data fitness and enable fast fine-tuning after pruning.
• Apply the method recursively across layers, starting from the first, to maintain consistency in feature map distributions.

Experimental results

Research questions

• RQ1 How can inter-layer dependency in channel redundancy be effectively modeled in Bayesian pruning?
• RQ2 Can a closed-form posterior approximation be achieved without incurring additional computational cost?
• RQ3 Does modeling dropout noise as a Markov chain improve pruning accuracy and model efficiency compared to independent channel assumptions?
• RQ4 To what extent can the proposed method reduce FLOPs while preserving model accuracy on large-scale datasets?
• RQ5 How does the method perform across different architectures like VGG16 and ResNet50 under varying pruning ratios?

Key findings

• RBP achieves up to 5.0× FLOPs reduction on VGG16 with only a 2.2% top-5 accuracy drop on ImageNet.
• For ResNet50, RBP reduces FLOPs by 2.2× with a 2.9% top-5 accuracy drop, demonstrating strong acceleration on deeper networks.
• The recursive variant, RRBP, removes 4,000 channels (more than RBP’s 3,824), showing improved pruning uniformity and robustness.
• RRBP achieves faster convergence during fine-tuning, with ResNet50-RRBP recovering performance by the 7th epoch.
• Dropout rates in the final residual block of ResNet50 remain away from 0 or 1, indicating sensitivity to pruning, consistent with experimental observations.
• The method maintains high data fitness post-pruning, enabling rapid performance recovery with only a few fine-tuning epochs.