[Paper Review] Building Efficient ConvNets using Redundant Feature Pruning

Babajide O. Ayinde, Jacek M. Żurada|arXiv (Cornell University)|Feb 21, 2018

Advanced Neural Network Applications21 references50 citations

TL;DR

The paper introduces an agglomerative clustering-based pruning method that removes redundant filters and their feature maps, reducing inference FLOPs by up to ~40% on CIFAR-10 models while maintaining competitive accuracy. Retraining after pruning helps recover performance without changing hyperparameters.

ABSTRACT

This paper presents an efficient technique to prune deep and/or wide convolutional neural network models by eliminating redundant features (or filters). Previous studies have shown that over-sized deep neural network models tend to produce a lot of redundant features that are either shifted version of one another or are very similar and show little or no variations; thus resulting in filtering redundancy. We propose to prune these redundant features along with their connecting feature maps according to their differentiation and based on their relative cosine distances in the feature space, thus yielding smaller network size with reduced inference costs and competitive performance. We empirically show on select models and CIFAR-10 dataset that inference costs can be reduced by 40% for VGG-16, 27% for ResNet-56, and 39% for ResNet-110.

Motivation & Objective

• Motivate reduction of network size and inference cost in over-parameterized CNNs by targeting redundant features (filters) that are highly similar in weight/activation space.
• Propose a one-shot pruning strategy that groups filters into clusters based on similarity and eliminates entire redundant clusters, along with their associated feature maps.
• Demonstrate that pruning can significantly lower FLOPs while preserving accuracy, with retraining required to recover performance.
• Provide practical guidance and empirical evidence on how redundancy-aware pruning compares with existing filter-pruning approaches.

Proposed method

• Represent each layer's filters as feature vectors (columns of W^(l)).
• Compute pairwise cosine similarity to define inter-filter distances.
• Apply hierarchical agglomerative clustering to merge the most similar filter groups until the average similarity between clusters falls below a threshold tau.
• Prune all filters except one representative per cluster (or prune enough filters to reach a desired prune count) along with corresponding feature maps and weights in the next layer.
• Update the kernel matrices for the l-th and (l+1)-th layers to reflect the reduced set of filters.
• Retrain the pruned network for a short period to recover accuracy without altering original hyperparameters.

Experimental results

Research questions

• RQ1 Can redundant filters in popular CNN architectures be identified and removed without degrading performance beyond retraining recovery?
• RQ2 How much inference cost (FLOPs) can be saved by pruning redundant features in VGG-16 and ResNet variants on CIFAR-10?
• RQ3 Does pruning based on feature redundancy outperform magnitude-based pruning methods like Li et al. (2017) on these models?
• RQ4 What is the impact of pruning order and the threshold tau on pruning effectiveness and final accuracy?
• RQ5 Is retraining from the pruned model sufficient to recover or improve accuracy, and how does this compare to pruning-from-scratch?

Key findings

• VGG-16 pruning with tau = 0.54 achieves >78% parameter pruning and ~40% FLOP reduction with competitive accuracy.
• ResNet-56 pruning achieves ~27.9% FLOP reduction with improved parameter pruning and accuracy after retraining compared to Li et al. (2017).
• ResNet-110 pruning yields ~39.1% FLOP reduction with similar parameter pruning, though accuracy after retraining may be slightly lower than Li et al. (2017) in some cases.
• Across models, pruning benefits from targeting later layers more aggressively, while early layers tend to be more sensitive to pruning and require cautious thresholds.
• Inference time reduction largely tracks FLOP reduction, with reported timing showing substantial speedups on CIFAR-10 test runs (Table 3).
• Pruning from a pretrained model and retraining typically yields better results than pruning from scratch (Table 4).