[Paper Review] Pruning Convolutional Neural Networks for Resource Efficient Inference

Pavlo Molchanov, Stephen Tyree+3 more|arXiv (Cornell University)|Nov 19, 2016

Domain Adaptation and Few-Shot Learning23 references1,204 citations

TL;DR

This paper introduces a Taylor-expansion-based pruning criterion for convolutional kernels, interleaving pruning with fine-tuning to create resource-efficient CNNs, validated on transfer learning tasks and large-scale ImageNet with substantial FLOPs reductions and modest accuracy loss.

ABSTRACT

We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with fine-tuning by backpropagation - a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters. We focus on transfer learning, where large pretrained networks are adapted to specialized tasks. The proposed criterion demonstrates superior performance compared to other criteria, e.g. the norm of kernel weights or feature map activation, for pruning large CNNs after adaptation to fine-grained classification tasks (Birds-200 and Flowers-102) relaying only on the first order gradient information. We also show that pruning can lead to more than 10x theoretical (5x practical) reduction in adapted 3D-convolutional filters with a small drop in accuracy in a recurrent gesture classifier. Finally, we show results for the large-scale ImageNet dataset to emphasize the flexibility of our approach.

Motivation & Objective

• Motivate reducing inference cost of CNNs through structured pruning of feature maps in transfer learning settings.
• Propose a fast, gradient-based saliency criterion to identify least important feature maps for pruning.
• Show that pruning interleaved with fine-tuning preserves generalization while reducing FLOPs and memory.
• Demonstrate scalability to large networks and datasets, including ImageNet, with real-speed improvements.

Proposed method

• Define pruning as an iterative, greedy process over feature maps with a pruning gate that sets maps to zero.
• Introduce saliency criteria including Taylor expansion-based pruning (Taylor criterion) that approximates the change in loss when pruning a map.
• Compare Taylor criterion against alternatives such as weight magnitude, activations, mutual information, and Optimal Brain Damage (OBD).
• Normalize criteria across layers to enable global pruning decisions.
• Optionally apply FLOPs regularization to bias pruning toward lower computational cost.
• Demonstrate that pruning can be interleaved with backpropagation-based fine-tuning to maintain accuracy.

Experimental results

Research questions

• RQ1 How can pruning decisions be made efficiently to minimize loss change while reducing network size and FLOPs?
• RQ2 Does a Taylor expansion-based saliency measure outperform other criteria for pruning CNN feature maps, especially in transfer learning?
• RQ3 What is the impact of pruning interleaved with fine-tuning on accuracy and speed across datasets and architectures?
• RQ4 Can FLOPs-aware pruning yield meaningful hardware-speedups on diverse platforms?

Key findings

• A Taylor expansion-based saliency criterion (Taylor criterion) achieves higher correlation with the oracle and better pruning performance than other criteria.
• Layer-wise normalization of pruning criteria improves cross-layer consistency and global pruning effectiveness.
• Iterative pruning with Taylor criterion yields substantial FLOPs reductions while preserving accuracy, outperforming alternatives in transfer learning tasks.
• Pruning 3D-CNNs for gesture recognition achieves around 12.6x GFLOPs reduction with only ~2.5% accuracy loss after fine-tuning.
• For ImageNet-scale networks, Taylor-based pruning combined with ample fine-tuning provides notable speedups and competitive top-5 accuracy after pruning.
• Practical speedups are demonstrated across multiple hardware setups, including CPU and GPU environments.