[Paper Review] Low precision storage for deep learning

Matthieu Courbariaux, Yoshua Bengio+1 more|arXiv (Cornell University)|Dec 22, 2014

Advanced Neural Network Applications23 references134 citations

TL;DR

This paper investigates low-precision storage formats—floating point, fixed point, and dynamic fixed point—for training and deploying Maxout neural networks on MNIST, CIFAR10, and SVHN. It demonstrates that 10-bit precision suffices for activations and gradients, and 12-bit for parameters, maintaining state-of-the-art performance, enabling efficient model storage and inference without significant accuracy loss.

ABSTRACT

We train a set of state of the art neural networks, the Maxout networks (Goodfellow et al., 2013a), on three benchmark datasets: the MNIST, CIFAR10 and SVHN, with three distinct storing formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those formats, we assess the impact of the precision of the storage on the final error of the training. We find that very low precision storage is sufficient not just for running trained networks but also for training them. For example, Maxout networks state-of-the-art results are nearly maintained with 10 bits for storing activations and gradients, and 12 bits for storing parameters.

Motivation & Objective

• To evaluate the impact of low-precision storage formats on deep learning model training and inference.
• To determine the minimum precision required for storing activations, gradients, and parameters without significant accuracy degradation.
• To compare fixed-point and dynamic fixed-point formats against standard floating-point representations in terms of model performance.
• To identify precision thresholds that preserve state-of-the-art results on standard benchmark datasets.

Proposed method

• Training Maxout networks on MNIST, CIFAR10, and SVHN using three storage formats: floating point, fixed point, and dynamic fixed point.
• Systematically varying the bit precision for storing activations, gradients, and parameters across multiple experiments.
• Using standard training procedures with stochastic gradient descent, while restricting weight, activation, and gradient storage to specified precision formats.
• Measuring final test error after training to assess the impact of precision on model performance.
• Employing dynamic fixed point to adapt the binary point position based on activation range, improving precision efficiency.

Experimental results

Research questions

• RQ1 What is the minimum precision required to store activations and gradients without degrading model accuracy during training?
• RQ2 How does fixed-point storage compare to floating-point in maintaining state-of-the-art performance on benchmark datasets?
• RQ3 Can dynamic fixed-point representation improve precision efficiency compared to standard fixed-point formats?
• RQ4 At what bit width do model performance losses become negligible for parameter, activation, and gradient storage?

Key findings

• Maxout networks achieve nearly state-of-the-art performance using only 10-bit precision for storing activations and gradients.
• Parameters can be stored with 12-bit precision while maintaining near-optimal test accuracy across all datasets.
• The use of dynamic fixed-point representation allows for better precision management without increasing bit width.
• Significant reductions in storage requirements are possible with minimal impact on model accuracy, especially for activations and gradients.