[Paper Review] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks

Zhonghui You, Kun Yan+3 more|arXiv (Cornell University)|Sep 18, 2019

Advanced Neural Network Applications111 citations

TL;DR

The paper introduces Gate Decorator for global filter pruning, plus Tick-Tock pruning and Group Pruning to accelerate CNNs without architectural changes, achieving state-of-the-art pruning ratios and accuracy gains on multiple datasets.

ABSTRACT

Filter pruning is one of the most effective ways to accelerate and compress convolutional neural networks (CNNs). In this work, we propose a global filter pruning algorithm called Gate Decorator, which transforms a vanilla CNN module by multiplying its output by the channel-wise scaling factors, i.e. gate. When the scaling factor is set to zero, it is equivalent to removing the corresponding filter. We use Taylor expansion to estimate the change in the loss function caused by setting the scaling factor to zero and use the estimation for the global filter importance ranking. Then we prune the network by removing those unimportant filters. After pruning, we merge all the scaling factors into its original module, so no special operations or structures are introduced. Moreover, we propose an iterative pruning framework called Tick-Tock to improve pruning accuracy. The extensive experiments demonstrate the effectiveness of our approaches. For example, we achieve the state-of-the-art pruning ratio on ResNet-56 by reducing 70% FLOPs without noticeable loss in accuracy. For ResNet-50 on ImageNet, our pruned model with 40% FLOPs reduction outperforms the baseline model by 0.31% in top-1 accuracy. Various datasets are used, including CIFAR-10, CIFAR-100, CUB-200, ImageNet ILSVRC-12 and PASCAL VOC 2011. Code is available at github.com/youzhonghui/gate-decorator-pruning

Motivation & Objective

• Motivate efficient deployment of CNNs on resource-constrained devices by reducing computation and storage.
• Develop a global filter importance ranking method that does not alter network design after pruning.
• Propose an iterative pruning framework (Tick-Tock) to improve pruning accuracy.
• Handle pruning constraints in networks with shortcut connections via Group Pruning.
• Demonstrate broad applicability across classification and segmentation tasks.

Proposed method

• Introduce Gate Decorator (GD) to estimate global filter importance via a Taylor expansion-based score Θ(φ_i) and gate scaling factors φ.
• Apply Gate Decorator to Batch Normalization by defining Gated Batch Normalization (GBN) to align filters with BN channels.
• Use the Tick-Tock pruning framework: Tick phase computes Θ on a subset of data with limited updatable parameters; Tock phase applies a sparse constraint to φ and fine-tunes.
• Incorporate Group Pruning to manage pruning in networks with shortcut connections by grouping BN blocks connected via pure shortcuts, summing their importance scores.
• Provide a pruning pipeline that can merge scaling factors after pruning to avoid extra architectural changes.
• Compare Gate Decorator with related GFIR methods (Slim, PCNN) and discuss differences in Taylor-based estimation, sparsity, and normalization considerations.

Experimental results

Research questions

• RQ1 Can a global filter importance ranking be reliably estimated without retraining the whole network?
• RQ2 Does the Gate Decorator enable higher pruning ratios with less accuracy loss compared to prior GFIR methods?
• RQ3 Can the Tick-Tock iterative pruning framework improve pruning accuracy over one-shot pruning?
• RQ4 How can pruning be performed on networks with shortcut connections without misalignment issues?
• RQ5 Is the Gate Decorator approach broadly applicable to different vision tasks beyond image classification?

Key findings

• Achieves state-of-the-art pruning on ResNet-56 with 70% FLOPs reduction with negligible accuracy loss on CIFAR-10.
• On ImageNet, pruned ResNet-50 with 40% FLOPs reduction outperforms the baseline by 0.31% top-1 accuracy.
• GBN-60 and GBN-50 on ImageNet improve inference speed over the baseline (e.g., 40% FLOPs pruning yields 1127 images/s vs 864 images/s in Titan X Pascal).
• Pruning FCN on PASCAL VOC 2011 maintains mIoU with substantial FLOPs and parameter reductions (27% FLOPs, 73% parameters) without accuracy loss.
• Tick-Tock pruning consistently outperforms One-Shot and Tick-Only, with higher final accuracy at the same FLOP targets, demonstrating the benefit of iterative pruning and sparse constraints.