[Paper Review] Low precision arithmetic for deep learning

Matthieu Courbariaux, Yoshua Bengio+1 more|arXiv (Cornell University)|Jan 1, 2014

Neural Networks and Applications87 citations

TL;DR

This paper investigates the feasibility of training deep neural networks using low-precision arithmetic, including floating-point, fixed-point, and dynamic fixed-point formats. It demonstrates that 10-bit precision for activations and gradients, and 12-bit for parameter updates, achieve near-state-of-the-art performance on MNIST, CIFAR10, and SVHN using Maxout networks, indicating that low-precision computation is sufficient for both training and inference.

ABSTRACT

We simulate the training of a set of state of the art neural networks, the Maxout networks (Goodfellow et al., 2013a), on three benchmark datasets: the MNIST, CIFAR10 and SVHN, with three distinct arithmetics: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those arithmetics, we assess the impact of the precision of the computations on the final error of the training. We find that very low precision computation is sufficient not just for running trained networks but also for training them. For example, almost state-of-the-art results were obtained on most datasets with 10 bits for computing activations and gradients, and 12 bits for storing updated parameters.

Motivation & Objective

• To evaluate the impact of low-precision arithmetic on the training of state-of-the-art deep neural networks.
• To determine the minimum precision required for both training and inference without significant accuracy loss.
• To compare the performance of floating-point, fixed-point, and dynamic fixed-point arithmetic in low-precision settings.
• To identify precision thresholds that maintain competitive test error rates across multiple benchmark datasets.

Proposed method

• Simulated training of Maxout networks on MNIST, CIFAR10, and SVHN using three arithmetic types: floating-point, fixed-point, and dynamic fixed-point.
• Varied the precision of computations (activations and gradients) and parameter storage across multiple bit-width configurations.
• Measured final test error after training to assess the impact of precision on model performance.
• Used consistent training hyperparameters across all precision settings to isolate the effect of numerical precision.
• Evaluated precision levels ranging from 8 to 16 bits for both computation and parameter storage.
• Employed dynamic fixed-point arithmetic to adapt the binary point position during computation, improving precision efficiency.

Experimental results

Research questions

• RQ1 What is the minimum precision required for training deep neural networks with acceptable accuracy?
• RQ2 How does low-precision arithmetic affect the final test error on standard benchmark datasets?
• RQ3 Can fixed-point or dynamic fixed-point arithmetic achieve performance comparable to full-precision floating-point training?
• RQ4 At what bit-width do the trade-offs between precision and computational efficiency become favorable for training?
• RQ5 Does the required precision vary across different datasets like MNIST, CIFAR10, and SVHN?

Key findings

• Training with 10-bit precision for activations and gradients achieved near-state-of-the-art performance on all three datasets.
• 12-bit precision for storing updated parameters was sufficient to maintain high model accuracy.
• Very low-precision arithmetic, including fixed-point and dynamic fixed-point, enabled training with minimal loss in final test error.
• The results indicate that precision can be significantly reduced without compromising model performance.
• The performance of low-precision training was consistent across MNIST, CIFAR10, and SVHN, suggesting broad applicability.
• Dynamic fixed-point arithmetic provided a practical balance between precision and efficiency in low-precision training.