[Paper Review] Deep Learning with Limited Numerical Precision

Suyog Gupta, Ankur Agrawal+2 more|arXiv (Cornell University)|Feb 9, 2015

Numerical Methods and Algorithms15 references1,075 citations

TL;DR

This paper demonstrates that deep neural networks can be trained using only 16-bit fixed-point arithmetic with stochastic rounding, achieving classification accuracy nearly identical to 32-bit floating-point training. The key contribution is a hardware-software co-design approach using a low-overhead FPGA accelerator that enables high-throughput, energy-efficient training by leveraging the noise tolerance of deep learning and stochastic rounding to mitigate precision loss.

ABSTRACT

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

Motivation & Objective

• To investigate whether deep neural networks can be trained using low-precision fixed-point arithmetic without significant accuracy degradation.
• To evaluate the impact of different rounding schemes—particularly stochastic rounding—on network training in low-precision settings.
• To design and prototype a hardware accelerator that supports low-precision fixed-point arithmetic with minimal area and power overhead.
• To enable energy-efficient, high-throughput deep learning training by co-optimizing algorithmic noise tolerance with low-level hardware approximation.

Proposed method

• The authors use 16-bit fixed-point representation for weights, activations, and gradients in deep neural network training.
• Stochastic rounding is applied during arithmetic operations to reduce systematic bias and improve convergence.
• A wavefront systolic array architecture is implemented on FPGA to perform matrix multiplications with high throughput.
• Stochastic rounding is implemented using a single DSP unit per output, leveraging LFSRs to generate random numbers for rounding.
• Overflow and underflow are detected using pattern detection in MSBs, with saturation applied when needed.
• The design uses compact, pipelined dataflow with local interconnects to minimize delay and maximize operating frequency.

Experimental results

Research questions

• RQ1 Can deep neural networks be trained effectively using only 16-bit fixed-point arithmetic?
• RQ2 How does the choice of rounding mode—specifically stochastic vs. conventional—impact training stability and accuracy?
• RQ3 What is the hardware overhead of integrating stochastic rounding into a fixed-point accelerator for deep learning?
• RQ4 Can a low-precision fixed-point accelerator achieve competitive throughput and energy efficiency compared to general-purpose processors and GPUs?

Key findings

• Deep networks trained with 16-bit fixed-point arithmetic and stochastic rounding achieve classification accuracy nearly identical to that of 32-bit floating-point training on MNIST and CIFAR10.
• The FPGA-based accelerator achieves a throughput of 260 G-ops/s at 7 W power consumption, resulting in 37 G-ops/s/W energy efficiency.
• The stochastic rounding module adds only 28 DSP units (less than 4% of total DSP resources) to the accelerator, demonstrating minimal hardware overhead.
• The wavefront systolic array design enables high throughput by allowing pipelined reuse of MAC units before previous operations complete.
• The system operates at 166 MHz on a Kintex-325T FPGA, outperforming general-purpose CPUs and GPUs in energy efficiency by a factor of 10–30.
• The results confirm that stochastic rounding is essential for low-precision training, as conventional rounding schemes lead to significant performance degradation.