[Paper Review] PruneNet: Channel Pruning via Global Importance

Ashish Khetan, Zohar Karnin|arXiv (Cornell University)|May 22, 2020

Advanced Neural Network Applications34 references11 citations

TL;DR

PruneNet proposes a data-driven, globally optimized channel pruning method that determines layer-specific channel widths via a lightweight optimization step, achieving superior accuracy and FLOP efficiency. On ImageNet, PruneNet's pruned ResNet-50 exceeds ResNet-34 and ResNet-18 in accuracy while reducing FLOPs, and pruned ResNet-101 outperforms ResNet-50.

ABSTRACT

Channel pruning is one of the predominant approaches for accelerating deep neural networks. Most existing pruning methods either train from scratch with a sparsity inducing term such as group lasso, or prune redundant channels in a pretrained network and then fine tune the network. Both strategies suffer from some limitations: the use of group lasso is computationally expensive, difficult to converge and often suffers from worse behavior due to the regularization bias. The methods that start with a pretrained network either prune channels uniformly across the layers or prune channels based on the basic statistics of the network parameters. These approaches either ignore the fact that some CNN layers are more redundant than others or fail to adequately identify the level of redundancy in different layers. In this work, we investigate a simple-yet-effective method for pruning channels based on a computationally light-weight yet effective data driven optimization step that discovers the necessary width per layer. Experiments conducted on ILSVRC-$12$ confirm effectiveness of our approach. With non-uniform pruning across the layers on ResNet-$50$, we are able to match the FLOP reduction of state-of-the-art channel pruning results while achieving a $0.98\%$ higher accuracy. Further, we show that our pruned ResNet-$50$ network outperforms ResNet-$34$ and ResNet-$18$ networks, and that our pruned ResNet-$101$ outperforms ResNet-$50$.

Motivation & Objective

• To address the limitations of uniform or statistics-based channel pruning by enabling layer-specific, data-driven channel width selection.
• To develop a computationally efficient method that avoids expensive regularization like group lasso while outperforming fine-tuned pruned models.
• To demonstrate that pruned versions of large networks (e.g., ResNet-50) can surpass smaller, hand-designed architectures (e.g., ResNet-34) in accuracy and efficiency.
• To validate that non-uniform pruning across layers yields better performance than uniform pruning or standard model variants.

Proposed method

• PruneNet introduces a global importance score based on a lightweight, data-driven optimization step to determine the optimal number of channels per layer.
• The method uses a differentiable, differentiable pruning strategy that jointly optimizes channel width and network weights during fine-tuning.
• It formulates channel pruning as a constrained optimization problem, where the objective is to minimize accuracy drop while reducing FLOPs.
• The approach uses a pruning schedule that adaptively removes channels based on their global contribution to the network’s performance.
• It leverages a two-stage training process: first, a global importance metric is computed using activation statistics; second, the network is fine-tuned with the selected channel widths.
• The method is applied post-pretraining and does not require retraining from scratch or complex regularization like group lasso.

Experimental results

Research questions

• RQ1 Can a data-driven, non-uniform channel pruning strategy outperform standard smaller architectures like ResNet-34 and ResNet-18?
• RQ2 Does a globally optimized channel pruning method achieve better accuracy-FLOP trade-offs than uniform or statistics-based pruning?
• RQ3 Can a pruned ResNet-50 surpass the performance of a larger, unpruned ResNet-101?
• RQ4 How does the performance of PruneNet compare to state-of-the-art channel pruning methods like DCP and ThiNet in terms of accuracy and efficiency?

Key findings

• Pruned ResNet-50 achieves 0.98% higher top-1 accuracy than state-of-the-art channel pruning methods while matching their FLOP reduction.
• The pruned ResNet-50 model outperforms both ResNet-34 and ResNet-18 in accuracy and throughput, despite having fewer FLOPs.
• Pruned ResNet-101 achieves higher accuracy and faster throughput than the original ResNet-50, demonstrating that pruning can create more efficient architectures than hand-designed smaller variants.
• On Caltech-256, pruned ResNet-50 with up to 10× FLOP reduction maintains or improves accuracy compared to the unpruned model, with a 7.53% top-1 accuracy gain at 5.75× FLOPs reduction.
• For aggressive pruning (e.g., 10× FLOPs reduction), PruneNet maintains a significant accuracy advantage over uniform pruning and state-of-the-art baselines like DCP and ThiNet.
• The GCP-f variant of PruneNet achieves a 3.86% accuracy gain over ResNet-50 at 5× FLOPs reduction, while reducing parameters by 3.09×.