[论文解读] Pruning Deep Neural Networks using Partial Least Squares

Artur Jordão, Kloss, Ricardo+2 more|arXiv (Cornell University)|Oct 17, 2018

Advanced Neural Network Applications参考文献 29被引用 7

一句话总结

本文提出了一种新颖的卷积神经网络滤波器剪枝方法，结合偏最小二乘法（PLS）与投影中变量重要性（VIP），基于滤波器与类别标签之间的判别关系识别并移除不重要的滤波器。该方法在不损失准确率的情况下实现高达67%的FLOPs减少，在损失可忽略的准确率下实现高达90%的FLOPs减少，通过有效特征选择与最小性能退化相结合，优于当前最先进方法。

ABSTRACT

Modern pattern recognition methods are based on convolutional networks since they are able to learn complex patterns that benefit the classification. However, convolutional networks are computationally expensive and require a considerable amount of memory, which limits their deployment on low-power and resource-constrained systems. To handle these problems, recent approaches have proposed pruning strategies that find and remove unimportant neurons (i.e., filters) in these networks. Despite achieving remarkable results, existing pruning approaches are ineffective since the accuracy of the original network is degraded. In this work, we propose a novel approach to efficiently remove filters from convolutional networks. Our approach estimates the filter importance based on its relationship with the class label on a low-dimensional space. This relationship is computed using Partial Least Squares (PLS) and Variable Importance in Projection (VIP). Our method is able to reduce up to 67% of the floating point operations (FLOPs) without penalizing the network accuracy. With a negligible drop in accuracy, we can reduce up to 90% of FLOPs. Additionally, sometimes the method is even able to improve the accuracy compared to original, unpruned, network. We show that employing PLS+VIP as the criterion for detecting the filters to be removed is better than recent feature selection techniques, which have been employed by state-of-the-art pruning methods. Finally, we show that the proposed method achieves the highest FLOPs reduction and the smallest drop in accuracy when compared to state-of-the-art pruning approaches. Codes are available at: https://github.com/arturjordao/PruningNeuralNetworks

研究动机与目标

• 为资源受限环境中的深度卷积神经网络的高计算与内存成本提供解决方案。
• 克服现有剪枝方法在降低网络准确率或需要高计算成本方面的局限性。
• 开发一种自动化、高效的剪枝准则，基于滤波器与类别标签之间的预测关系识别不重要的滤波器。
• 在保持或甚至提升网络准确率的同时，实现更优的FLOPs减少。

提出的方法

• 通过在网络中输入训练数据，将每个卷积滤波器的输出表示为特征向量。
• 从网络中所有滤波器输出构建高维特征空间。
• 应用偏最小二乘法（PLS），将高维空间投影到低维潜在空间，以最大化特征与类别标签之间的协方差。
• 使用投影中变量重要性（VIP）量化每个滤波器对潜在空间的贡献，识别不重要的滤波器。
• 迭代移除VIP得分较低的滤波器，重复此过程直至达到所需的FLOPs减少目标。
• 剪枝后对网络进行微调以优化性能。

实验结果

研究问题

• RQ1 像PLS这样的判别性特征选择方法能否有效识别深度网络中的不重要滤波器？
• RQ2 将VIP用作滤波器重要性准则，是否能带来优于L1范数或学习型剪枝代理的剪枝性能？
• RQ3 基于PLS+VIP的剪枝能否在更低的准确率退化下实现比当前最先进方法更高的FLOPs减少？
• RQ4 与原始未剪枝网络相比，所提方法是否能提升或保持准确率？

主要发现

• 所提方法在ImageNet和CIFAR-10上实现高达67%的FLOPs减少，且准确率无任何下降。
• 在VGG16和ResNet架构上，准确率损失小于1%的情况下，实现高达90%的FLOPs减少。
• 在ResNet56和ResNet110上，该方法实现的FLOPs减少量是Huang et al. [5]的1.5倍，且准确率下降更小。
• 在VGG16上，该方法实现的FLOPs减少量是所有对比方法的1.8倍，且准确率下降最小。
• 该方法从高FLOPs层（如第2–10层）中移除超过50%的滤波器，而现有方法多集中于低FLOPs层。
• 在某些情况下，剪枝后的网络甚至比原始未剪枝模型准确率更高。