[论文解读] Towards Unified INT8 Training for Convolutional Neural Network

Feng Zhu, Ruihao Gong+6 more|arXiv (Cornell University)|Dec 29, 2019

Advanced Neural Network Applications参考文献 55被引用 20

一句话总结

本文提出了一种针对卷积神经网络的统一INT8训练框架，通过两种新技术——方向敏感梯度裁剪（Direction Sensitive Gradient Clipping）和偏差对抗学习率调节（Deviation Counteractive Learning Rate Scaling）——稳定低比特反向传播。该框架在多种网络结构（包括MobileNetV2、InceptionV3和目标检测器）上实现了稳定且精确的INT8训练，且在未使用专用硬件的Pascal GPU上将训练时间减少了22%。

ABSTRACT

Recently low-bit (e.g., 8-bit) network quantization has been extensively studied to accelerate the inference. Besides inference, low-bit training with quantized gradients can further bring more considerable acceleration, since the backward process is often computation-intensive. Unfortunately, the inappropriate quantization of backward propagation usually makes the training unstable and even crash. There lacks a successful unified low-bit training framework that can support diverse networks on various tasks. In this paper, we give an attempt to build a unified 8-bit (INT8) training framework for common convolutional neural networks from the aspects of both accuracy and speed. First, we empirically find the four distinctive characteristics of gradients, which provide us insightful clues for gradient quantization. Then, we theoretically give an in-depth analysis of the convergence bound and derive two principles for stable INT8 training. Finally, we propose two universal techniques, including Direction Sensitive Gradient Clipping that reduces the direction deviation of gradients and Deviation Counteractive Learning Rate Scaling that avoids illegal gradient update along the wrong direction. The experiments show that our unified solution promises accurate and efficient INT8 training for a variety of networks and tasks, including MobileNetV2, InceptionV3 and object detection that prior studies have never succeeded. Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, and reduces the training time by 22% on Pascal GPU without too much optimization effort. We believe that this pioneering study will help lead the community towards a fully unified INT8 training for convolutional neural networks.

研究动机与目标

• 解决由量化梯度引起的INT8训练不稳定与崩溃问题。
• 开发一种统一的、与硬件无关的框架，支持多种CNN架构与任务。
• 通过收敛界分析，为稳定INT8训练提供理论依据。
• 实现在无需定制硬件或复杂网络修改的前提下，利用市售GPU实现实际加速。

提出的方法

• 通过实验识别出梯度的四种独特特性：陡峭且宽广、演化性、深度相关性与结构相关性，这些特性使其与权重和激活值显著不同。
• 理论上推导出收敛界，建立梯度量化误差与学习率之间的关联，从而提出稳定训练的两条原则。
• 提出方向敏感梯度裁剪，通过基于梯度幅值与方向的自适应裁剪，最小化量化梯度中的方向偏差。
• 提出偏差对抗学习率调节，通过动态调整学习率来抵消量化偏差的负面影响。
• 利用NVIDIA的DP4A指令集在Pascal GPU上实现该框架，确保与市售硬件的兼容性。
• 仅对卷积层进行INT8优化，以极小的工程投入实现显著加速。

实验结果

研究问题

• RQ1 梯度具有哪些独特特性，使得INT8训练相较于权重/激活量化更具挑战性？
• RQ2 在梯度量化误差存在的情况下，INT8训练的收敛性如何实现理论上的边界约束？
• RQ3 能否使梯度裁剪具备方向敏感性，以减少训练过程中的方向偏差？
• RQ4 能否设计一种学习率调节机制，以抵消量化偏差的负面影响？
• RQ5 是否可能在不依赖专用硬件的前提下，实现对多种网络与任务的稳定且高效的INT8训练？

主要发现

• 所提出的统一INT8训练框架在ImageNet上使用InceptionV3实现了95.00%的top-1准确率，与全精度性能相当。
• 对于MobileNetV2和InceptionV3，该方法在INT8训练中实现了71.20%的top-1准确率，仅比FP32模型低1.19%。
• 在目标检测任务中，该方法在PASCAL VOC和COCO数据集上的mAP均保持在全精度模型的1.8%以内。
• 该框架在GeForce GTX 1080Ti（Pascal GPU）上将端到端训练时间减少了22%，反向传播速度提升了1.94倍。
• 该方法是首个成功实现Faster R-CNN和RetinaNet等目标检测网络INT8训练的方案。
• 仅对ResNet-50的卷积层进行INT8优化，即可实现前向传播1.63倍、反向传播1.94倍的加速，且优化工作量极小。