[Paper Review] Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers

Yukuan Yang, Shuang Wu+4 more|arXiv (Cornell University)|Sep 5, 2019

Advanced Neural Network Applications34 references11 citations

TL;DR

This paper proposes WAGEUBN, the first complete 8-bit integer quantization framework for training large-scale deep neural networks, enabling full integer arithmetic across all data paths—including weights, activations, gradients, errors, updates, batch normalization, and the Momentum optimizer. It achieves competitive ImageNet accuracy with up to 4× memory reduction, 9× faster computation, and over 30× lower power consumption compared to FP32, enabling efficient online learning on portable devices.

ABSTRACT

Deep neural network (DNN) quantization converting floating-point (FP) data in the network to integers (INT) is an effective way to shrink the model size for memory saving and simplify the operations for compute acceleration. Recently, researches on DNN quantization develop from inference to training, laying a foundation for the online training on accelerators. However, existing schemes leaving batch normalization (BN) untouched during training are mostly incomplete quantization that still adopts high precision FP in some parts of the data paths. Currently, there is no solution that can use only low bit-width INT data during the whole training process of large-scale DNNs with acceptable accuracy. In this work, through decomposing all the computation steps in DNNs and fusing three special quantization functions to satisfy the different precision requirements, we propose a unified complete quantization framework termed as ``WAGEUBN'' to quantize DNNs involving all data paths including W (Weights), A (Activation), G (Gradient), E (Error), U (Update), and BN. Moreover, the Momentum optimizer is also quantized to realize a completely quantized framework. Experiments on ResNet18/34/50 models demonstrate that WAGEUBN can achieve competitive accuracy on the ImageNet dataset. For the first time, the study of quantization in large-scale DNNs is advanced to the full 8-bit INT level. In this way, all the operations in the training and inference can be bit-wise operations, pushing towards faster processing speed, decreased memory cost, and higher energy efficiency. Our throughout quantization framework has great potential for future efficient portable devices with online learning ability.

Motivation & Objective

• To address incomplete quantization in DNN training, where only partial data paths are quantized or FP precision is retained in critical components like batch normalization and optimizers.
• To enable full 8-bit integer computation throughout the entire training pipeline, including forward and backward passes, weight updates, and optimization, for improved hardware efficiency.
• To overcome non-convergence issues in low-precision training by introducing a flag-bit-based quantization method that expands dynamic range and preserves small-value information.
• To extend quantization to large-scale models like ResNet18/34/50 on ImageNet, demonstrating feasibility and performance at the full 8-bit integer level.
• To provide a hardware-optimized framework that enables faster, lower-power, and memory-efficient training for future portable devices with online learning capabilities.

Proposed method

• The framework decomposes all DNN computation steps into distinct data paths: W (weights), A (activations), G (gradients), E (errors), U (updates), BN (batch normalization), and O (optimizer), each quantized independently.
• Three specialized quantization functions are fused to meet varying precision demands: standard 8-bit quantization, flag-bit-enhanced 8-bit quantization for error and gradient tensors, and a scaled integer representation for batch normalization.
• A flag bit is introduced in the quantization of error and gradient tensors to extend the effective dynamic range, preserving small but informative values and preventing non-convergence.
• Batch normalization layers are replaced with learnable scaling factors in INT8, eliminating FP dependency while maintaining training stability.
• The Momentum optimizer is fully quantized into 8-bit integers, enabling all optimization steps to be performed using bit-wise operations.
• The framework is evaluated on ResNet18/34/50 using ImageNet, with end-to-end training and inference fully in 8-bit integer arithmetic.

Experimental results

Research questions

• RQ1 Can full 8-bit integer arithmetic be used throughout the entire DNN training process, including backpropagation and optimization, without sacrificing model accuracy?
• RQ2 How can the dynamic range of low-precision quantization be expanded to prevent non-convergence, especially for small-value gradients and errors?
• RQ3 Can batch normalization be effectively replaced with a quantized, fixed-scale equivalent in 8-bit integer format without degrading training performance?
• RQ4 What are the actual hardware efficiency gains—speed, power, and area—of full 8-bit integer training compared to mixed-precision or FP32 training?
• RQ5 Can this framework be successfully applied to large-scale models like ResNet50 on ImageNet while maintaining competitive accuracy?

Key findings

• WAGEUBN achieves competitive top-1 accuracy of 70.2% on ImageNet using ResNet18, demonstrating that full 8-bit integer training is feasible for large-scale models.
• The framework reduces model size by approximately 4× compared to full-precision FP32 networks, significantly lowering memory requirements.
• Multiplication and accumulation operations in WAGEUBN are over 3× and 9× faster, respectively, compared to FP32, with power consumption reduced by over 30×.
• The flag-bit quantization method increases the data coverage ratio of small-value gradients and errors, preventing non-convergence and improving training stability.
• Hardware simulations on FPGA show that INT8 operations achieve 9× smaller circuit area, 10× lower power, and 3× higher speed than FP32 for multiplication, and 30× lower power and 9× faster speed for accumulation.
• This work establishes the first complete 8-bit integer training framework for large-scale DNNs, enabling bit-wise operations across all training components and paving the way for efficient on-device online learning.