[论文解读] ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

Jian-Hao Luo, Jianxin Wu+1 more|arXiv (Cornell University)|Jul 20, 2017

Advanced Neural Network Applications参考文献 29被引用 104

一句话总结

ThiNet 引入了一种基于过滤器级 pruning 框架，使用下一层的统计信息来决定哪些过滤器需要被剪裁，从而在 VGG-16、ResNet-50 等 ImageNet 模型上实现同时加速和压缩，且精度损失最小。

ABSTRACT

We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31$ imes$ FLOPs reduction and 16.63$ imes$ compression on VGG-16, with only 0.52$\%$ top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1$\%$ top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability.

研究动机与目标

• 推动在资源受限设备上部署的 CNN 模型压缩。
• 提出一个统一框架（ThiNet），通过过滤器剪裁实现同时加速和压缩。
• 将过滤器剪裁形式化定义为基于下一层统计信息的优化问题。
• 在大规模基准（ImageNet）和迁移学习场景中展示有效性。

提出的方法

• 在不改变网络结构的前提下进行过滤器级别剪裁，以实现与现成库的兼容。
• 谨慎地基于下一层信息来剪裁过滤器，以近似其输出。
• 收集与下一层输入输出相关的训练样本以指导剪裁。
• 将通道选择表述为一个组合优化问题，通过贪心算法求解。
• 可选地通过最小二乘重权重化来改进初始化以便于微调。
• 剪裁后进行微调以恢复性能并逐层迭代。

实验结果

研究问题

• RQ1 给定层的剪裁决策能否可靠地由下一层的统计信息而非当前层引导？
• RQ2 基于数据驱动的贪心通道选择方法在识别不重要过滤器方面有多高的效果？
• RQ3 ThiNet 剪裁在模型大小、FLOPs 及在像 ImageNet 这样的基准和迁移学习任务中的影响如何？

主要发现

Model	Top-1	Top-5	#Param.	#FLOPs	f./b. (ms)
原始	68.34%	88.44%	138.34M	30.94B	189.92/407.56
ThiNet-Conv	69.80%	89.53%	131.44M	9.58B	76.71/152.05
Train from scratch	67.00%	87.45%	131.44M	9.58B	76.71/152.05
ThiNet-GAP	67.34%	87.92%	8.32M	9.34B	71.73/145.51
ThiNet-Tiny	59.34%	81.97%	1.32M	2.01B	29.51/55.83

• 在 VGG-16 上，ThiNet 实现了 3.31× 的 FLOPs 减少和 16.63× 的压缩，顶级5准确率下降仅为 0.52%。
• 在 ResNet-50 上，ThiNet 将参数和 FLOPs 减半以上，顶级5准确率下降约 1%。
• 剪裁后的 VGG-16 可以被压缩至 5.05 MB，同时保持与 AlexNet 相当的准确性，在迁移任务中显示出更好的泛化能力。
• ThiNet-Tiny 实现了显著的参数减少（1.32M）并具备有竞争力的准确性，使非常紧凑的模型成为可能。
• 与其他剪裁标准相比，ThiNet 在相同或更大压缩比下获得了更高的准确性。