[Paper Review] Network Pruning using Adaptive Exemplar Filters

Mingbao Lin, Rongrong Ji+5 more|arXiv (Cornell University)|Jan 20, 2021

Advanced Neural Network Applications55 references4 citations

TL;DR

EPruner proposes a data-independent, CPU-efficient filter pruning method using adaptive exemplar filters selected via Affinity Propagation on weight matrices. By automatically identifying informative filters without human-designed rules or costly retraining, it achieves up to 76.34% FLOPs reduction on VGGNet-16 with 0.06% accuracy gain and 65.12% FLOPs reduction on ResNet-152 with only 0.71% top-5 accuracy drop.

ABSTRACT

Popular network pruning algorithms reduce redundant information by optimizing hand-crafted models, and may cause suboptimal performance and long time in selecting filters. We innovatively introduce adaptive exemplar filters to simplify the algorithm design, resulting in an automatic and efficient pruning approach called EPruner. Inspired by the face recognition community, we use a message passing algorithm Affinity Propagation on the weight matrices to obtain an adaptive number of exemplars, which then act as the preserved filters. EPruner breaks the dependency on the training data in determining the "important" filters and allows the CPU implementation in seconds, an order of magnitude faster than GPU based SOTAs. Moreover, we show that the weights of exemplars provide a better initialization for the fine-tuning. On VGGNet-16, EPruner achieves a 76.34%-FLOPs reduction by removing 88.80% parameters, with 0.06% accuracy improvement on CIFAR-10. In ResNet-152, EPruner achieves a 65.12%-FLOPs reduction by removing 64.18% parameters, with only 0.71% top-5 accuracy loss on ILSVRC-2012. Our code can be available at https://github.com/lmbxmu/EPruner.

Motivation & Objective

• To eliminate reliance on hand-crafted rules and data-dependent search in structured filter pruning.
• To reduce pruning time from hours (GPU-based SOTAs) to seconds using CPU-only computation.
• To identify a robust, adaptive method for selecting the most informative filters without retraining or hyperparameter tuning.
• To provide better initialization for fine-tuning by inheriting exemplar filter weights.
• To enable efficient, automatic, and scalable model compression across diverse CNN architectures.

Proposed method

• Treat each filter in a convolutional layer as a high-dimensional data point in a similarity graph.
• Apply the Affinity Propagation message-passing algorithm to the weight matrices to identify adaptive exemplars without human-defined thresholds.
• Use the number of exemplars determined by the algorithm itself, eliminating the need for manual specification of pruning ratio.
• Preserve only the exemplar filters and their corresponding channels, discarding the rest.
• Inherit the weights of the exemplars as initialization for fine-tuning, leveraging their pre-trained representational power.
• Implement the entire pruning pipeline on CPU, achieving sub-second inference time for architecture selection.

Experimental results

Research questions

• RQ1 Can a data-independent, automatic method identify the optimal number of filters to preserve without human-designed rules?
• RQ2 Can the Affinity Propagation algorithm effectively select high-impact filters from pre-trained CNNs for structured pruning?
• RQ3 Does using exemplar filter weights as initialization lead to better fine-tuning performance than random or norm-based initialization?
• RQ4 Can the pruning process be accelerated to seconds on CPU while maintaining or improving accuracy?
• RQ5 How does the method scale across different network architectures and datasets?

Key findings

• On VGGNet-16 with CIFAR-10, EPruner achieves a 76.34% reduction in FLOPs by removing 88.80% of parameters, with a 0.06% accuracy improvement over the original model.
• On ResNet-152 with ImageNet-ILSVRC-2012, EPruner reduces FLOPs by 65.12% and parameters by 64.18%, with only a 0.71% top-5 accuracy drop.
• The method completes architecture selection in under 13 seconds on CPU for ResNet-152, compared to over 75,000 seconds on GPUs using ABCPruner.
• EPruner’s exemplar-based initialization consistently outperforms random and ℓ₁-norm-based initialization in fine-tuning, demonstrating superior warm-up quality.
• The influence of the hyperparameter β is minimal within a stable range (β ∈ [0.72, 0.73]), enabling reliable and predictable pruning without extensive tuning.
• The method is robust across multiple architectures (VGG, ResNet, GoogLeNet) and datasets, showing consistent performance and efficiency gains.