[Paper Review] ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

Jian-Hao Luo, Jianxin Wu+1 more|arXiv (Cornell University)|Jul 20, 2017

Advanced Neural Network ApplicationsComputer Science29 references104 citations

TL;DR

ThiNet introduces a filter-level pruning framework that uses statistics from the next layer to decide which filters to prune, enabling simultaneous acceleration and compression with minimal accuracy loss on ImageNet models like VGG-16 and ResNet-50.

ABSTRACT

We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31$ imes$ FLOPs reduction and 16.63$ imes$ compression on VGG-16, with only 0.52$\%$ top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1$\%$ top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability.

Motivation & Objective

• Motivate CNN model compression for deployment on resource-constrained devices.
• Propose a unified framework (ThiNet) for simultaneous acceleration and compression via filter pruning.
• Formally define filter pruning as an optimization problem based on next-layer statistics.
• Demonstrate effectiveness on large-scale benchmarks (ImageNet) and transfer learning scenarios.

Proposed method

• Prune at the filter level without changing network structure, enabling compatibility with off-the-shelf libraries.
• Prudence: prune filters based on information from the next layer to approximate its outputs.
• Collect training examples that relate next-layer inputs to outputs to guide pruning.
• Formulate channel selection as a combinatorial optimization problem solved via a greedy algorithm.
• Optionally refine pruning with a least-squares reweighting to improve initialization for fine-tuning.
• Fine-tune after pruning to recover performance and iterate layer by layer.

Experimental results

Research questions

• RQ1 Can pruning decisions for a given layer be reliably guided by statistics from the next layer rather than the current layer?
• RQ2 How effective is a data-driven, greedy channel selection approach for identifying unimportant filters?
• RQ3 What is the impact of ThiNet pruning on model size, FLOPs, and accuracy on large-scale benchmarks like ImageNet and transfer learning tasks?

Key findings

• On VGG-16, ThiNet achieves 3.31× FLOPs reduction and 16.63× compression with only 0.52% top-5 accuracy drop.
• On ResNet-50, ThiNet reduces more than half of parameters and FLOPs with about 1% top-5 accuracy drop.
• Pruned VGG-16 can be reduced to 5.05 MB while preserving AlexNet-level accuracy and showing improved generalization in transfer tasks.
• ThiNet-Tiny achieves substantial parameter reduction (1.32M) with competitive accuracy, enabling very compact models.
• Compared with other pruning criteria, ThiNet yields higher accuracy at similar or greater compression.