[Paper Review] Pruning Filters for Efficient ConvNets

Hao Li, Asim Kadav+3 more|arXiv (Cornell University)|Aug 31, 2016

Advanced Neural Network Applications28 references685 citations

TL;DR

The paper presents a method to prune whole convolutional filters (and their feature maps) to reduce CNN inference cost without introducing sparse connectivity, achieving substantial FLOP reductions with minimal accuracy loss.

ABSTRACT

The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.

Motivation & Objective

• Motivate reducing CNN inference costs while preserving accuracy.
• Propose a structured, one-shot filter pruning method that removes entire filters and their feature maps.
• Show that pruning filters avoids irregular sparsity and works with dense BLAS libraries.
• Provide guidance on pruning sensitivity across layers and across network blocks (e.g., ResNet blocks).
• Demonstrate practical reductions in FLOPs on VGG-16 and ResNet architectures with retraining.

Proposed method

• Compute per-filter importance using the L1 norm of the filter weights and prune the lowest-weight filters in each layer.
• Remove the corresponding feature maps and kernels in the following layer to maintain consistency.
• Adopt a one-shot pruning and retraining strategy across multiple layers to save retraining time.
• Analyze layer and stage sensitivity to pruning to decide pruning rates and skip pruning for sensitive layers.
• For ResNets, prune in a way that respects residual/shortcut connections, using projection shortcut pruning rules.
• Compare independent vs greedy pruning approaches for multi-layer pruning to assess holistic effects.

Experimental results

Research questions

• RQ1 Can whole-filter pruning reduce FLOPs significantly without large accuracy loss across common CNNs?
• RQ2 Which filters (by what criterion) are least impactful on output accuracy when pruned?
• RQ3 How does pruning across multiple layers (and across stages) affect network performance compared to layer-wise pruning?
• RQ4 What are the practical considerations for pruning residual blocks in ResNets while preserving shortcut integrity?
• RQ5 How does filter-level pruning compare to activation-based feature-map pruning in terms of effectiveness and data dependence?

Key findings

• Pruning small-magnitude filters reduces FLOPs by up to 34% on VGG-16 and up to 38% on ResNet-110 on CIFAR-10 with near-original accuracy after retraining.
• L1-norm based filter ranking outperforms random or largest-filter pruning across layers for CIFAR-10 on VGG-16.
• Many layers and stages tolerate substantial pruning, with deeper stages often being more resilient to pruning.
• Pruning residual blocks in ResNet can be done with targeted strategies (e.g., pruning based on shortcut projections) to preserve accuracy while reducing compute.
• One-shot pruning with subsequent retraining can recover much of the lost accuracy, reducing training time compared with iterative layer-by-layer pruning.
• Activation-based feature-map pruning using data-dependent statistics generally underperforms simple L1-based filter pruning in this study.