[Paper Review] BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Matthieu Courbariaux, Yoshua Bengio+1 more|PolyPublie (École Polytechnique de Montréal)|Nov 2, 2015

Advanced Neural Network Applications44 references1,829 citations

TL;DR

BinaryConnect trains deep neural networks using binary weights (+1 or -1) during forward and backward propagations, while maintaining full-precision weights for gradient accumulation. The method achieves near state-of-the-art accuracy on MNIST, CIFAR-10, and SVHN, demonstrating that binary weights with high-precision gradient updates enable efficient training suitable for specialized hardware with minimal accuracy loss.

ABSTRACT

Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.

Motivation & Objective

• To enable efficient training and inference of deep neural networks on low-power devices by using binary weights during forward and backward passes.
• To reduce computational complexity and hardware cost by replacing multiply-accumulate operations with simple additions.
• To maintain high model accuracy despite binary weight constraints through high-precision gradient accumulation.
• To explore whether binary weights act as a regularizer, improving generalization in deep networks.
• To demonstrate feasibility of training with binary weights on standard vision benchmarks without full-precision weight updates during propagation.

Proposed method

• BinaryConnect constrains weights to ±1 during forward and backward propagations, replacing multiply-accumulate operations with additions.
• The real-valued stored weights are updated using stochastic gradient descent with gradients accumulated in high-precision form.
• Binarization is performed via a deterministic sign function: w_b = sign(w), where w is the real-valued weight.
• The method leverages the noise tolerance of stochastic gradient descent, where weight discretization acts as a form of regularization.
• Gradients are computed and backpropagated using the binary weights, but the stored weights are updated using full-precision gradients.
• The approach is compatible with batch normalization and optimizers like ADAM, and is implemented using Theano and Pylearn2.

Experimental results

Research questions

• RQ1 Can deep neural networks be trained effectively using only binary weights during forward and backward propagation?
• RQ2 Does the use of binary weights during inference and backpropagation act as a regularizer, improving generalization?
• RQ3 To what extent can model accuracy be preserved when replacing full-precision multiply-accumulate operations with binary additions?
• RQ4 How does BinaryConnect compare to other quantization methods like ternary weights or expectation backpropagation in terms of accuracy and hardware efficiency?
• RQ5 Can the method be extended to convolutional networks and larger datasets like CIFAR-10 and SVHN without significant performance degradation?

Key findings

• BinaryConnect achieves test error rates of 1.44% on permutation-invariant MNIST, approaching the state-of-the-art.
• On CIFAR-10, BinaryConnect attains a test error rate of 12.0%, which is near state-of-the-art for binary-weight methods.
• On SVHN, the method achieves a test error rate of 11.7%, demonstrating strong performance on a more complex dataset.
• The method reduces the number of multiplications by approximately two-thirds during training, significantly improving computational efficiency.
• The use of binary weights during propagation, combined with high-precision stored weights, enables stable training and good generalization.
• The approach is compatible with standard deep learning techniques such as batch normalization, ReLU activations, and ADAM optimization.