[Paper Review] SCOP: Scientific Control for Reliable Neural Network Pruning

Yehui Tang, Yunhe Wang+5 more|arXiv (Cornell University)|Oct 21, 2020

Advanced Neural Network Applications46 references85 citations

TL;DR

SCOP introduces knockoff features as a scientific control group to reliably prune filters in pre-trained CNNs, achieving substantial compression with minimal accuracy loss (e.g., 57.8% parameters and 60.2% FLOPs reduced on ResNet-101 with 0.01% top-1 loss).

ABSTRACT

This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Existing pruning methods have developed various hypotheses to approximate the importance of filters to the network and then execute filter pruning accordingly. To increase the reliability of the results, we prefer to have a more rigorous research design by including a scientific control group as an essential part to minimize the effect of all factors except the association between the filter and expected network output. Acting as a control group, knockoff feature is generated to mimic the feature map produced by the network filter, but they are conditionally independent of the example label given the real feature map. We theoretically suggest that the knockoff condition can be approximately preserved given the information propagation of network layers. Besides the real feature map on an intermediate layer, the corresponding knockoff feature is brought in as another auxiliary input signal for the subsequent layers. Redundant filters can be discovered in the adversarial process of different features. Through experiments, we demonstrate the superiority of the proposed algorithm over state-of-the-art methods. For example, our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet. The code is available at https://github.com/huawei-noah/Pruning/tree/master/SCOP_NeurIPS2020.

Motivation & Objective

• Motivate reliable filter pruning by minimizing interference from irrelevant factors in pruning decisions.
• Propose a scientific control framework using knockoff features to distinguish truly important filters.
• Develop an efficient method to generate knockoff data and integrate it into pruning with learnable scaling factors.
• Demonstrate state-of-the-art compression-accuracy trade-offs on CIFAR-10 and ImageNet datasets.

Proposed method

• Define knockoff features that are exchangeable with real features but independent of labels.
• Show that knockoff features can be propagated through network layers via two lemmas, enabling layer-wise knockoffs without retraining generators.
• Introduce an adversarial selection layer that jointly scales real and knockoff features with a constraint beta + beta~ = 1.
• Optimize scaling factors for real and knockoff features using labeled data to identify filters that rely more on real data than on knockoffs.
• Prune filters by computing an importance statistic I = beta - beta~ (adjusted with BN scales when present).
• Fine-tune the pruned network to recover or improve performance.

Experimental results

Research questions

• RQ1 Can knockoff features serve as an effective control group to identify redundant CNN filters during pruning?
• RQ2 How can knockoff data be efficiently generated and propagated through deep networks to support reliable pruning?
• RQ3 Do pruning results using SCOP improve the accuracy-parameter/FLOPs trade-off compared with state-of-the-art methods on standard benchmarks?
• RQ4 Is the proposed SCOP framework robust across architectures and datasets (CIFAR-10 and ImageNet)?

Key findings

• SCOP reduces parameters and FLOPs substantially while preserving accuracy better than several state-of-the-art pruning methods on CIFAR-10 (e.g., ResNet-56: 56.0% FLOPs reduction with 0.06% accuracy drop).
• On ImageNet, SCOP achieves competitive top-1/top-5 accuracy with larger compression than many baselines (e.g., ResNet-50: 24.74% top-1 with notable FLOPs reduction).
• SCOP can reach 57.8% parameter reduction and 60.2% FLOPs reduction for ResNet-101 with only 0.01% top-1 accuracy loss.
• Ablation studies show that removing the control group increases error, confirming the importance of knockoffs for reliable pruning.
• Realistic acceleration on GPUs aligns with substantial theoretical FLOPs reductions, validating practical benefits of SCOP.