[Paper Review] Training deep neural networks with low precision multiplications

Matthieu Courbariaux, Yoshua Bengio+1 more|PolyPublie (École Polytechnique de Montréal)|Dec 22, 2014

Numerical Methods and Algorithms18 references302 citations

TL;DR

This paper demonstrates that deep neural networks can be successfully trained using extremely low-precision multipliers—down to 9-bit fixed-point or 10-bit dynamic fixed-point arithmetic—while maintaining high accuracy. By using high-precision accumulators and higher-precision weight updates, the authors achieve near-state-of-the-art performance on MNIST, CIFAR-10, and SVHN using minimal arithmetic precision, enabling efficient hardware deployment.

ABSTRACT

Multipliers are the most space and power-hungry arithmetic operators of the digital implementation of deep neural networks. We train a set of state-of-the-art neural networks (Maxout networks) on three benchmark datasets: MNIST, CIFAR-10 and SVHN. They are trained with three distinct formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those formats, we assess the impact of the precision of the multiplications on the final error after training. We find that very low precision is sufficient not just for running trained networks but also for training them. For example, it is possible to train Maxout networks with 10 bits multiplications.

Motivation & Objective

• To investigate whether extremely low-precision multipliers can be used not only for inference but also for end-to-end training of deep neural networks.
• To evaluate the impact of precision reduction on training stability and final test error across floating-point, fixed-point, and dynamic fixed-point formats.
• To determine the minimum bit-widths required for activations and weight updates that preserve model accuracy.
• To propose and validate a dynamic fixed-point format that adapts scaling factors per layer to improve precision efficiency.
• To enable energy- and area-efficient hardware designs for deep learning by minimizing multiplier precision without sacrificing performance.

Proposed method

• Training Maxout networks on MNIST, CIFAR-10, and SVHN using three arithmetic formats: floating point, fixed point, and dynamic fixed point.
• Using high-precision accumulators (32-bit) to maintain numerical stability during forward and backward passes despite low-precision multipliers.
• Applying higher-precision arithmetic (19–20 bits) for weight updates compared to lower-precision (9–11 bits) for activations and propagations.
• Employing dynamic fixed point with adaptive scaling factors learned during initial training and updated every 10,000 examples.
• Using stochastic rounding and normalization techniques to reduce quantization error in low-precision settings.
• Evaluating generalization error across multiple datasets and precision levels to identify precision thresholds.

Experimental results

Research questions

• RQ1 Can deep neural networks be trained using 10-bit or lower precision multiplications without significant accuracy degradation?
• RQ2 How does dynamic fixed point compare to fixed point and floating point in terms of precision efficiency and training stability?
• RQ3 What is the minimum bit-width required for activations and weight updates to maintain acceptable test error rates?
• RQ4 Does using higher-precision arithmetic during weight updates significantly improve training convergence in low-precision settings?
• RQ5 Can low-precision training be effectively applied to modern deep learning models like Maxout networks on standard benchmarks?

Key findings

• Training with 10-bit fixed-point multipliers results in minimal accuracy degradation, with test error rates comparable to full-precision training on MNIST, CIFAR-10, and SVHN.
• The minimum bit-width for propagations in fixed point is 19 bits (20 with sign), below which test error increases sharply.
• For dynamic fixed point, 9-bit propagations and 11-bit weight updates achieve near-optimal performance, with only a slight increase in error on SVHN.
• Using higher-precision accumulators and higher-precision weight updates significantly improves training stability and final accuracy in low-precision settings.
• Dynamic fixed point outperforms standard fixed point by adapting scaling factors per layer, reducing quantization error and enabling lower bit-widths.
• The use of half-precision floating point (16-bit) has little to no impact on training, suggesting that even 16-bit floating point is sufficient for training.