[Paper Review] Training Deep Neural Network in Limited Precision

Hyunsun Park, Jun Haeng Lee+3 more|arXiv (Cornell University)|Oct 12, 2018

Advanced Neural Network Applications28 references4 citations

TL;DR

This paper proposes a lazy update mechanism using Kahan summation to enable stable training of deep neural networks in low precision (e.g., INT8) by accumulating small gradients in an auxiliary accumulator, preventing precision loss during parameter updates. It also introduces a bit-width guideline for the final fully connected layer before softmax based on class count. The method achieves full-precision accuracy on CIFAR-10, ImageNet, GANs, and LSTM tasks using only 8-bit weights and 16-bit activations, gradients, and optimizers.

ABSTRACT

Energy and resource efficient training of DNNs will greatly extend the applications of deep learning. However, there are three major obstacles which mandate accurate calculation in high precision. In this paper, we tackle two of them related to the loss of gradients during parameter update and backpropagation through a softmax nonlinearity layer in low precision training. We implemented SGD with Kahan summation by employing an additional parameter to virtually extend the bit-width of the parameters for a reliable parameter update. We also proposed a simple guideline to help select the appropriate bit-width for the last FC layer followed by a softmax nonlinearity layer. It determines the lower bound of the required bit-width based on the class size of the dataset. Extensive experiments on various network architectures and benchmarks verifies the effectiveness of the proposed technique for low precision training.

Motivation & Objective

• Address the challenge of unstable parameter updates in low-precision DNN training due to insufficient gradient precision.
• Overcome the numerical instability introduced by softmax nonlinearity in low-precision computation.
• Enable full-precision training accuracy using only low-precision hardware (e.g., 8-bit) without requiring mixed-precision or specialized hardware.
• Provide a practical guideline for selecting minimum bit-width of the final fully connected layer based on dataset class size.
• Demonstrate end-to-end low-precision training on diverse architectures and benchmarks without accuracy degradation.

Proposed method

• Employ Kahan summation in stochastic gradient descent (SGD) by introducing an auxiliary accumulator to accumulate small gradients over multiple steps.
• Update the main network parameters only when the accumulated gradient value is large enough to affect the parameter within the target precision (e.g., 8-bit), reducing precision loss.
• Use a dynamic fixed-point number system with 8-bit weights and 16-bit accumulators to virtually extend the effective bit-width of parameters.
• Apply the lazy update mechanism to both weight and bias updates in all layers, including the final fully connected layer before softmax.
• Propose a bit-width guideline: the final FC layer should have at least log₂(C) bits, where C is the number of output classes, to prevent accuracy degradation.
• Implement the method on a simple 8-bit fixed-point accelerator without requiring specialized hardware or stochastic quantization.

Experimental results

Research questions

• RQ1 Can stable parameter updates be achieved in low-precision DNN training using only standard integer arithmetic and no high-precision hardware?
• RQ2 How can the numerical instability introduced by the softmax nonlinearity in low-precision training be mitigated?
• RQ3 What is the minimum required bit-width for the final fully connected layer before softmax to maintain training accuracy in low-precision settings?
• RQ4 Can full-precision accuracy be achieved in low-precision training across diverse architectures (CNNs, GANs, LSTMs) and benchmarks (CIFAR-10, ImageNet, TIDIGITS) using only 8-bit weights and 16-bit activations and gradients?
• RQ5 Does the proposed lazy update mechanism outperform standard low-precision training in terms of convergence and final accuracy?

Key findings

• The proposed lazy update with Kahan summation enables stable training of DNNs in 8-bit precision with full-precision accuracy on CIFAR-10, achieving 93.4% test accuracy with INT8 weights and INT16 activations, gradients, and optimizers.
• On ImageNet, the method achieved 75.3% top-1 accuracy using only 8-bit weights and 16-bit activations, gradients, and optimizers—comparable to full-precision training.
• For GANs, the method successfully generated high-quality 64×64 images of faces and bedrooms using only INT8 weights and INT16 for all other tensors, with no visible degradation.
• On the TIDIGITS dataset, LSTM training in INT8 with the lazy update achieved 97.41% validation accuracy, improving from 75.21% without the method and matching the 97.62% of FP32 training.
• The proposed bit-width guideline for the final FC layer (log₂(C) bits) effectively prevents accuracy degradation and provides a lower bound for safe low-precision training.
• The method enables end-to-end training of diverse DNNs (CNNs, GANs, LSTMs) in low precision without requiring mixed-precision hardware or specialized operations, achieving competitive results across all benchmarks.