[论文解读] Structural Compression of Convolutional Neural Networks Based on Greedy Filter Pruning

Reza Abbasi-Asl, Bin Yu|arXiv (Cornell University)|May 20, 2017

Advanced Neural Network Applications参考文献 12被引用 33

一句话总结

本文提出一种基于分类准确率降低（CAR）指数的贪心滤波器剪枝方法，用于压缩卷积神经网络（CNN）。该方法通过迭代移除滤波器来实现，依据其对分类准确率的影响进行排序。与先前方法相比，该方法在准确率上表现更优——在AlexNet的第一和第二层分别剪枝一半滤波器时，准确率分别高出最佳基准26%和20%；同时通过微调实现42倍的模型尺寸压缩，且性能接近原始模型。

ABSTRACT

Convolutional neural networks (CNNs) have state-of-the-art performance on many problems in machine vision. However, networks with superior performance often have millions of weights so that it is difficult or impossible to use CNNs on computationally limited devices or to humanly interpret them. A myriad of CNN compression approaches have been proposed and they involve pruning and compressing the weights and filters. In this article, we introduce a greedy structural compression scheme that prunes filters in a trained CNN. We define a filter importance index equal to the classification accuracy reduction (CAR) of the network after pruning that filter (similarly defined as RAR for regression). We then iteratively prune filters based on the CAR index. This algorithm achieves substantially higher classification accuracy in AlexNet compared to other structural compression schemes that prune filters. Pruning half of the filters in the first or second layer of AlexNet, our CAR algorithm achieves 26% and 20% higher classification accuracies respectively, compared to the best benchmark filter pruning scheme. Our CAR algorithm, combined with further weight pruning and compressing, reduces the size of first or second convolutional layer in AlexNet by a factor of 42, while achieving close to original classification accuracy through retraining (or fine-tuning) network. Finally, we demonstrate the interpretability of CAR-compressed CNNs by showing that our algorithm prunes filters with visually redundant functionalities. In fact, out of top 20 CAR-pruned filters in AlexNet, 17 of them in the first layer and 14 of them in the second layer are color-selective filters as opposed to shape-selective filters. To our knowledge, this is the first reported result on the connection between compression and interpretability of CNNs.

研究动机与目标

• 解决在计算资源受限设备上部署大型高性能CNN的挑战。
• 在保持分类准确率的同时，超越现有结构化剪枝方法，进一步提升模型压缩效果。
• 通过分析剪枝滤波器的功能角色，探索压缩后CNN的可解释性。
• 通过实证分析，建立模型压缩与滤波器可解释性之间的关联。

提出的方法

• 该方法将滤波器重要性指数定义为剪枝某一特定滤波器所导致的分类准确率降低（CAR）值。
• 以贪心策略迭代剪枝，优先移除导致CAR最小的滤波器。
• 剪枝后对网络进行微调（fine-tuning）以恢复准确率。
• 该方法结合结构化剪枝与后续的权重重剪枝及压缩，实现进一步的模型尺寸缩减。
• 利用CAR指数对滤波器进行排序，优先移除对准确率影响最小的滤波器。
• 该方法应用于AlexNet，并在第一和第二卷积层进行了消融实验。

实验结果

研究问题

• RQ1 基于CAR的贪心滤波器剪枝策略是否能在分类准确率上超越现有结构化压缩方法？
• RQ2 滤波器剪枝在多大程度上可实现模型尺寸的显著缩减，同时保持高准确率？
• RQ3 压缩后的CNN中，被剪枝的滤波器是否表现出冗余或可解释的功能？
• RQ4 是否存在可测量的关联，将结构化压缩与模型可解释性的提升联系起来？

主要发现

• 在AlexNet的第一层剪枝一半滤波器时，采用CAR方法实现的分类准确率比最佳基准剪枝方案高出26%。
• 在AlexNet的第二层剪枝一半滤波器时，准确率比最佳基准方法高出20%。
• 经过CAR压缩的网络，其第一或第二卷积层的尺寸缩小了42倍，且在微调后保持了接近原始的分类准确率。
• 在第一层中，前20名CAR剪枝滤波器中有17个具有颜色选择性，表明冗余滤波器已被有效移除。
• 在第二层中，前20名CAR剪枝滤波器中有14个具有颜色选择性，进一步支持了对功能冗余滤波器的移除。
• 本工作首次通过实证识别出具有视觉冗余功能的剪枝滤波器，提供了结构压缩与CNN模型可解释性提升之间关联的直接证据。