[Paper Review] REPrune: Filter Pruning via Representative Election

Min-Cheol Park, Woojeong Kim+1 more|arXiv (Cornell University)|Jul 14, 2020

Internet Traffic Analysis and Secure E-voting39 references4 citations

TL;DR

REPrune proposes a novel filter pruning method that selects representative filters via clustering instead of relying on norm-based criteria, improving model compression and accuracy recovery. By using agglomerative clustering and the silhouette coefficient to identify informative filters, REPrune reduces FLOPs by over 49% with a 0.53% accuracy gain on ResNet-110 for CIFAR-10 and achieves 41.8% FLOPs reduction with 1.67% lower Top-1 loss on ResNet-18 for ImageNet.

ABSTRACT

Even though norm-based filter pruning methods are widely accepted, it is questionable whether the "smaller-norm-less-important" criterion is optimal in determining filters to prune. Especially when we can keep only a small fraction of the original filters, it is more crucial to choose the filters that can best represent the whole filters regardless of norm values. Our novel pruning method entitled "REPrune" addresses this problem by selecting representative filters via clustering. By selecting one filter from a cluster of similar filters and avoiding selecting adjacent large filters, REPrune can achieve a better compression rate with similar accuracy. Our method also recovers the accuracy more rapidly and requires a smaller shift of filters during fine-tuning. Empirically, REPrune reduces more than 49% FLOPs, with 0.53% accuracy gain on ResNet-110 for CIFAR-10. Also, REPrune reduces more than 41.8% FLOPs with 1.67% Top-1 validation loss on ResNet-18 for ImageNet.

Motivation & Objective

• Address the limitation of norm-based filter pruning, which assumes smaller norm implies less importance, especially under high pruning ratios.
• Improve model compression and accuracy retention by selecting filters that best represent the distribution of filters in each layer.
• Reduce the shift in filter weights during fine-tuning by preserving filters that maintain structural similarity to the original model.
• Enable faster convergence of validation accuracy during fine-tuning by selecting high-quality, representative filters.
• Demonstrate effectiveness on highly optimized models like ResNet under extreme pruning constraints.

Proposed method

• Apply agglomerative clustering to group filters based on their feature similarity within each convolutional layer.
• Use the silhouette coefficient to evaluate cluster quality and identify the most representative filter in each cluster.
• Select the filter closest to the cluster centroid as the representative, ensuring it best captures the cluster's distribution.
• Avoid selecting adjacent large-norm filters by enforcing diversity in selected representatives across clusters.
• Perform dynamic pruning per layer, preserving more filters in high-level layers where computation is most expensive.
• Fine-tune the pruned model with minimal weight shift due to the high cosine similarity between original and pruned filters.

Experimental results

Research questions

• RQ1 Can filter selection based on distributional similarity outperform norm-based criteria under extreme pruning ratios?
• RQ2 Does selecting representative filters via clustering lead to faster accuracy recovery during fine-tuning compared to norm-based methods?
• RQ3 To what extent does the cosine similarity between pre- and post-fine-tuning filters remain high when using distribution-aware pruning?
• RQ4 How does REPrune perform in terms of FLOPs reduction and accuracy retention on standard benchmarks like CIFAR-10 and ImageNet?
• RQ5 Can REPrune maintain model performance on highly optimized architectures such as ResNet under aggressive pruning?

Key findings

• REPrune reduces FLOPs by more than 49% on ResNet-110 for CIFAR-10 while achieving a 0.53% accuracy gain compared to the original model.
• On ResNet-18 for ImageNet, REPrune reduces FLOPs by 41.8% and achieves a 1.67% lower Top-1 validation loss than the baseline.
• The pruned model using REPrune recovers validation accuracy 4x faster than norm-based methods, reaching in 10 epochs what others take 40 epochs to achieve.
• Cosine similarity between filters before and after fine-tuning is significantly higher in REPrune (average ~0.4–0.6) than in SFP and FPGM (70%+ similarity in -0.1 to 0.1 range).
• REPrune maintains a blue-toned heatmap of cosine similarity across layers, indicating consistent filter orientation preservation, unlike the white/red patterns of norm-based methods.
• The method achieves up to 8.84x inference speedup with only 2.6% accuracy drop on ResNet-56 for CIFAR-100, even with 83.3% parameter reduction.