[Paper Review] Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN

Xiaohan Ding, Tianxiang Hao+3 more|arXiv (Cornell University)|Jul 30, 2021

Advanced Neural Network Applications90 references4 citations

TL;DR

This paper proposes Centripetal SGD (C-SGD), a novel training method that induces identical filter redundancy in deep CNNs by clustering filters and driving them to converge to the same values during backpropagation. By creating purely redundant filters, C-SGD enables one-step, accuracy-preserving pruning without finetuning, achieving state-of-the-art performance on CIFAR-10, ImageNet, and ResNet/DenseNet architectures with minimal computational overhead.

ABSTRACT

The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops. However, the training objective of CNNs usually tends to minimize an accuracy-related loss function without any attention paid to the redundancy, making the redundancy distribute randomly on all the filters, such that removing any of them may trigger information loss and accuracy drop, necessitating a following finetuning step for recovery. In this paper, we propose to manipulate the redundancy during training to facilitate network pruning. To this end, we propose a novel Centripetal SGD (C-SGD) to make some filters identical, resulting in ideal redundancy patterns, as such filters become purely redundant due to their duplicates; hence removing them does not harm the network. As shown on CIFAR and ImageNet, C-SGD delivers better performance because the redundancy is better organized, compared to the existing methods. The efficiency also characterizes C-SGD because it is as fast as regular SGD, requires no finetuning, and can be conducted simultaneously on all the layers even in very deep CNNs. Besides, C-SGD can improve the accuracy of CNNs by first training a model with the same architecture but wider layers then squeezing it into the original width.

Motivation & Objective

• To address the challenge of constrained filter pruning in very deep and complex CNNs with skip connections and dense blocks, where standard pruning methods risk degrading performance due to non-uniform filter importance.
• To eliminate the need for finetuning after pruning by creating purely redundant filters that can be removed without performance loss.
• To improve the efficiency and effectiveness of filter pruning by organizing redundancy into ideal, identical-filter patterns rather than relying on small-norm regularization.
• To enable simultaneous, global pruning across all layers in very deep networks, overcoming limitations of layer-by-layer or filter-by-filter pruning strategies.
• To explore whether structured redundancy can enhance training convergence and model performance, proposing a new training paradigm—Scaling and Squeezing—for performance boosting.

Proposed method

• C-SGD introduces a modified loss function that enforces filter clustering by minimizing the sum of squared deviations between filters within each cluster.
• It uses a learnable averaging matrix $\bm{\Gamma}$ and a decaying matrix $\bm{\Lambda}$ to compute the centroid update for each filter cluster during backpropagation.
• The method applies a centripetal force via a hyperparameter $\epsilon$, which drives filters in each cluster toward their mean value at a constant rate, ensuring convergence to identical weights.
• The algorithm is integrated into standard SGD training and requires only two additional matrix multiplications per layer per iteration, incurring negligible computational cost.
• The redundancy pattern is created during training, so pruning can be performed immediately after training without any finetuning step.
• The approach supports end-to-end training and pruning, enabling simultaneous pruning of all layers in deep networks like ResNet-164 and DenseNet-121.

Experimental results

Research questions

• RQ1 Can inducing identical filter redundancy during training enable one-step, lossless pruning without finetuning in deep CNNs?
• RQ2 How does C-SGD perform on complex architectures like ResNet and DenseNet, where constrained pruning is challenging due to skip connections and dense blocks?
• RQ3 Does the proposed redundancy pattern (identical filters) outperform small-norm redundancy (e.g., group-Lasso) in terms of pruning efficiency and accuracy retention?
• RQ4 Can C-SGD be applied globally across all layers simultaneously, and does it maintain or improve model accuracy in very deep networks?
• RQ5 Does the training strategy of Scaling and Squeezing—training a wider model and then pruning to original width—benefit from C-SGD and improve final accuracy?

Key findings

• C-SGD enables one-step pruning on ResNet-164 and DenseNet-121 with no accuracy drop, even when pruning all layers simultaneously, demonstrating its robustness in very deep networks.
• On CIFAR-10, C-SGD achieves 76.25% top-1 accuracy after pruning ResNet-56 to 2× width, outperforming prior methods that require finetuning and suffer accuracy loss.
• On ImageNet, C-SGD achieves 75.88% top-1 accuracy after pruning ResNet-56 to 2× width, surpassing baseline methods that show higher error increases (0.94 vs. 0.74) under similar settings.
• The method is computationally efficient: training speed is indistinguishable from standard SGD, and it requires no finetuning, unlike most prior approaches.
• C-SGD is robust to the centripetal strength $\epsilon$, with convergence and performance stable across $\epsilon = 1\times 10^{-3}$, $2\times 10^{-3}$, and $1\times 10^{-2}$.
• The Scaling and Squeezing strategy—training a wider model with C-SGD and then pruning to original width—improves final accuracy, suggesting that redundancy can aid convergence and generalization.