[Paper Review] Towards Unified INT8 Training for Convolutional Neural Network

Feng Zhu, Ruihao Gong+6 more|arXiv (Cornell University)|Dec 29, 2019

Advanced Neural Network ApplicationsComputer Science55 references20 citations

TL;DR

This paper proposes a unified INT8 training framework for convolutional neural networks that stabilizes low-bit backpropagation through two novel techniques: Direction Sensitive Gradient Clipping and Deviation Counteractive Learning Rate Scaling. It achieves stable, accurate INT8 training across diverse networks—including MobileNetV2, InceptionV3, and object detectors—while reducing training time by 22% on Pascal GPUs without specialized hardware.

ABSTRACT

Recently low-bit (e.g., 8-bit) network quantization has been extensively studied to accelerate the inference. Besides inference, low-bit training with quantized gradients can further bring more considerable acceleration, since the backward process is often computation-intensive. Unfortunately, the inappropriate quantization of backward propagation usually makes the training unstable and even crash. There lacks a successful unified low-bit training framework that can support diverse networks on various tasks. In this paper, we give an attempt to build a unified 8-bit (INT8) training framework for common convolutional neural networks from the aspects of both accuracy and speed. First, we empirically find the four distinctive characteristics of gradients, which provide us insightful clues for gradient quantization. Then, we theoretically give an in-depth analysis of the convergence bound and derive two principles for stable INT8 training. Finally, we propose two universal techniques, including Direction Sensitive Gradient Clipping that reduces the direction deviation of gradients and Deviation Counteractive Learning Rate Scaling that avoids illegal gradient update along the wrong direction. The experiments show that our unified solution promises accurate and efficient INT8 training for a variety of networks and tasks, including MobileNetV2, InceptionV3 and object detection that prior studies have never succeeded. Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, and reduces the training time by 22% on Pascal GPU without too much optimization effort. We believe that this pioneering study will help lead the community towards a fully unified INT8 training for convolutional neural networks.

Motivation & Objective

• To address the instability and crash issues in INT8 training caused by quantized gradients.
• To develop a unified, hardware-agnostic framework that supports diverse CNN architectures and tasks.
• To provide theoretical justification for stable INT8 training through convergence bound analysis.
• To enable practical speedup on off-the-shelf GPUs without requiring custom hardware or complex network modifications.

Proposed method

• Empirically identify four unique characteristics of gradients: sharp and wide, evolutionary, depth-specific, and structure-specific, which differentiate them from weights and activations.
• Theoretically derive a convergence bound that links gradient quantization error and learning rate, establishing two principles for stable training.
• Propose Direction Sensitive Gradient Clipping to minimize direction deviation in quantized gradients by adaptively clipping based on gradient magnitude and direction.
• Introduce Deviation Counteractive Learning Rate Scaling to counteract the negative impact of quantization deviation by dynamically adjusting the learning rate.
• Implement the framework using NVIDIA's DP4A instruction set on Pascal GPUs to ensure compatibility with off-the-shelf hardware.
• Optimize only the convolutional layers for INT8, achieving significant speedup with minimal engineering effort.

Experimental results

Research questions

• RQ1 What are the distinct characteristics of gradients that make INT8 training more challenging than weight/activation quantization?
• RQ2 How can the convergence of INT8 training be theoretically bounded under gradient quantization error?
• RQ3 Can gradient clipping be made direction-sensitive to reduce directional deviation during training?
• RQ4 Can learning rate scaling be designed to counteract the negative effects of quantization deviation?
• RQ5 Is it possible to achieve stable and efficient INT8 training across diverse networks and tasks without specialized hardware?

Key findings

• The proposed unified INT8 training framework achieves 95.00% top-1 accuracy on ImageNet using InceptionV3, matching full-precision performance.
• For MobileNetV2 and InceptionV3, the method achieves 71.20% top-1 accuracy in INT8 training, with only a 1.19% drop from FP32.
• On object detection tasks, the method maintains mAP within 1.8% of full-precision models on both PASCAL VOC and COCO datasets.
• The framework reduces end-to-end training time by 22% on GeForce GTX 1080Ti (Pascal GPU), with 1.94× speedup in backward pass.
• The method is the first to successfully enable INT8 training for object detection networks like Faster R-CNN and RetinaNet.
• The solution achieves 1.63× speedup in forward pass and 1.94× in backward pass on ResNet-50 using only INT8 convolutional layers with minimal optimization.