[Paper Review] An Entropy-based Pruning Method for CNN Compression

Jian-Hao Luo, Jianxin Wu|arXiv (Cornell University)|Jun 19, 2017

Advanced Neural Network Applications31 references147 citations

TL;DR

The paper proposes an entropy-based filter pruning method to accelerate and compress CNNs, reducing intermediate activations and maintaining accuracy with fine-tuning; achieves significant speed-ups and compression on VGG-16 and ResNet-50.

ABSTRACT

This paper aims to simultaneously accelerate and compress off-the-shelf CNN models via filter pruning strategy. The importance of each filter is evaluated by the proposed entropy-based method first. Then several unimportant filters are discarded to get a smaller CNN model. Finally, fine-tuning is adopted to recover its generalization ability which is damaged during filter pruning. Our method can reduce the size of intermediate activations, which would dominate most memory footprint during model training stage but is less concerned in previous compression methods. Experiments on the ILSVRC-12 benchmark demonstrate the effectiveness of our method. Compared with previous filter importance evaluation criteria, our entropy-based method obtains better performance. We achieve 3.3x speed-up and 16.64x compression on VGG-16, 1.54x acceleration and 1.47x compression on ResNet-50, both with about 1% top-5 accuracy decrease.

Motivation & Objective

• Motivate reducing both compute and memory footprint of CNNs for deployment on resource-constrained devices.
• Introduce an entropy-based criterion to evaluate filter importance via activation channel information content.
• Develop a pruning strategy that can be applied during training and inference without specialized libraries.
• Propose a learning schedule to balance pruning speed and final model accuracy.
• Demonstrate effectiveness on VGG-16 and ResNet-50 using ImageNet (ILSVRC-12).

Proposed method

• Compute channel-wise entropy of activation channels after global average pooling to estimate filter importance.
• Rank filters by entropy and prune a fixed compression rate by removing low-entropy channels and corresponding next-layer channels.
• Fine-tune the pruned network to recover accuracy and adjust the learning schedule to speed up training.
• Prune different architectures with tailored strategies (e.g., prune initial convolutional layers in VGG-16; prune middle layers in ResNet blocks).
• Compare against baselines (Pruning, APoZ, Taylor expansion) and assess domain adaptation on CUB-200-2011 and Indoor-67.

Experimental results

Research questions

• RQ1 Can entropy-based activation information effectively identify unimportant filters for pruning in CNNs?
• RQ2 How does entropy-based pruning influence speed, parameter count, and intermediate activation memory on standard CNNs?
• RQ3 Can pruned networks maintain competitive accuracy after fine-tuning on ImageNet and transfer to other domains?
• RQ4 What learning schedule best trades off pruning speed and final accuracy across architectures like VGG-16 and ResNet-50?

Key findings

• On VGG-16, pruning with entropy-based selection achieves 3.3x speed-up and 16.64x compression with approximately 1% top-5 accuracy loss.
• On ResNet-50, the method yields 1.54x acceleration and 1.47x compression with about 1% top-5 accuracy loss.
• Entropy-based pruning compares favorably to APoZ and Taylor-based criteria and can outperform some baselines while maintaining compatibility with off-the-shelf libraries.
• A learning schedule that prunes layer-by-layer with brief fine-tuning between steps dramatically reduces training time while preserving performance compared to naive iterative pruning.