[Paper Review] Training Deep Neural Networks with 8-bit Floating Point Numbers

Naigang Wang, Jungwook Choi+3 more|arXiv (Cornell University)|Dec 19, 2018

Neural Networks and Applications205 citations

TL;DR

The paper demonstrates successful training of various DNNs using 8-bit floating point numbers (FP8) with FP16 accumulations and FP16 weight updates, enabled by chunk-based accumulation and floating point stochastic rounding, achieving comparable accuracy to FP32 baselines while reducing memory and compute requirements.

ABSTRACT

The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems.

Motivation & Objective

• Motivate reducing training precision to 8 bits without sacrificing model accuracy.
• Introduce FP8/FP16 formats and techniques to overcome accumulation and update challenges.
• Showcase wide empirical validation across CNNs and DNNs on standard datasets.
• Propose hardware-efficient approaches enabling 2-4x improvements in throughput and energy efficiency.

Proposed method

• Define FP8 (1,5,2) and FP16 (1,6,9) formats for data and accumulations.
• Use chunk-based accumulation to partition long dot-products and reduce swamping errors.
• Apply floating point stochastic rounding to weight updates to preserve information loss during rounding.
• Maintain FP16 for last-layer GEMMs to stabilize Softmax computations.
• Adopt loss-scaling to preserve small gradients during backpropagation.
• Validate via emulated reduced-precision experiments across multiple networks and datasets.

Experimental results

Research questions

• RQ1 Can 8-bit floating point representations be used to train DNNs without accuracy loss across diverse models and datasets?
• RQ2 How can swamping and accumulation errors be mitigated when using low-precision formats during training?
• RQ3 What are the practical hardware implications of FP8 training in terms of memory, bandwidth, and energy efficiency?
• RQ4 What role do first and last layer precisions play in FP8 training success?
• RQ5 How do rounding modes affect FP8 training accuracy?

Key findings

• FP8 training with FP16 accumulations and FP16 weight updates achieves comparable test accuracy to FP32 baselines across CIFAR-10 CNN, CIFAR-10 ResNet, BN50-DNN, AlexNet, ResNet-18, and ResNet-50.
• Memory for weights and master copies is reduced by ~2× due to FP8 weights and FP16 master copies.
• Chunk-based accumulation and floating point stochastic rounding effectively mitigate swamping, enabling robust 8-bit training.
• Loss-scaling and reserving FP16 for the last layer GEMMs stabilize training on large datasets like ImageNet.
• Nearest rounding degrades accuracy; stochastic rounding maintains baseline performance during FP16 weight updates.
• Hardware demonstrations suggest FP8 engines can be 2-4× more energy-efficient than FP16 equivalents.