[论文解读] Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning

Tien-Ju Yang, Yu‐Hsin Chen+1 more|arXiv (Cornell University)|Nov 16, 2016

Advanced Neural Network Applications参考文献 15被引用 31

一句话总结

本文提出了一种针对卷积神经网络（CNNs）的节能剪枝算法，该算法使用硬件实测参数直接优化能耗。通过基于输出特征图误差逐层剪枝，并采用闭式最小二乘法微调，该方法在AlexNet上将能耗降低3.7倍，在GoogLeNet上降低1.6倍，且top-5准确率损失低于1%。

ABSTRACT

Deep convolutional neural networks (CNNs) are indispensable to state-of-the-art computer vision algorithms. However, they are still rarely deployed on battery-powered mobile devices, such as smartphones and wearable gadgets, where vision algorithms can enable many revolutionary real-world applications. The key limiting factor is the high energy consumption of CNN processing due to its high computational complexity. While there are many previous efforts that try to reduce the CNN model size or amount of computation, we find that they do not necessarily result in lower energy consumption, and therefore do not serve as a good metric for energy cost estimation. To close the gap between CNN design and energy consumption optimization, we propose an energy-aware pruning algorithm for CNNs that directly uses energy consumption estimation of a CNN to guide the pruning process. The energy estimation methodology uses parameters extrapolated from actual hardware measurements that target realistic battery-powered system setups. The proposed layer-by-layer pruning algorithm also prunes more aggressively than previously proposed pruning methods by minimizing the error in output feature maps instead of filter weights. For each layer, the weights are first pruned and then locally fine-tuned with a closed-form least-square solution to quickly restore the accuracy. After all layers are pruned, the entire network is further globally fine-tuned using back-propagation. With the proposed pruning method, the energy consumption of AlexNet and GoogLeNet are reduced by 3.7x and 1.6x, respectively, with less than 1% top-5 accuracy loss. Finally, we show that pruning the AlexNet with a reduced number of target classes can greatly decrease the number of weights but the energy reduction is limited. Energy modeling tool and energy-aware pruned models available at this http URL

研究动机与目标

• 为解决在电池供电设备上部署CNN时模型压缩与实际能效之间的差距。
• 开发一种直接以能耗为优化目标的剪枝方法，而非模型大小或FLOPs。
• 通过最小化能耗，在移动和可穿戴设备上实现CNN的高效部署，同时保持准确率。
• 为移动系统上的CNN推理创建一种实用且基于硬件的能耗估算模型。

提出的方法

• 该方法使用硬件实测参数估算每层的能耗，从而实现直接的能耗感知优化。
• 基于输出特征图的误差而非滤波器权重大小，进行逐层剪枝。
• 剪枝后，使用闭式最小二乘解对每一层进行微调，以快速恢复准确率。
• 随后通过反向传播对整个网络进行全局微调，进一步提升性能。
• 该方法优先考虑能耗降低而非模型大小减少，采用基于真实移动硬件的定制能耗估算模型。

实验结果

研究问题

• RQ1 基于实际能耗估算的剪枝是否能带来更高效的移动设备CNN部署？
• RQ2 在剪枝过程中最小化输出特征图误差是否比传统的基于权重大小的剪枝更具能耗效率？
• RQ3 结合逐层剪枝与闭式微调的方法，在降低能耗的同时能否有效保持准确率？
• RQ4 减少目标类别数量在多大程度上影响剪枝模型的能耗节省？

主要发现

• 使用该方法，AlexNet的能耗降低了3.7倍，且top-5准确率损失低于1%。
• GoogLeNet实现了1.6倍的能耗降低，同时保持了高准确率。
• 与传统剪枝方法相比，该节能剪枝方法在能耗效率方面表现更优，即使未最小化模型大小。
• 减少目标类别数显著减少了参数量，但能耗节省有限，表明类别数量本身并不能决定能耗效率。
• 闭式微调步骤能够快速恢复剪枝后的准确率，减少了对大规模训练的需求。