[Paper Review] Binarized Neural Networks

Itay Hubara, Daniel Soudry+1 more|arXiv (Cornell University)|Feb 8, 2016

Advanced Neural Network Applications919 citations

TL;DR

The paper trains Binarized Neural Networks with binary weights and activations, achieving near state-of-the-art results on MNIST, CIFAR-10, and SVHN, while greatly reducing memory/computation and enabling a 7x speedup for MNIST with a binary GEMM kernel.

ABSTRACT

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time and when computing the parameters' gradient at train-time. We conduct two sets of experiments, each based on a different framework, namely Torch7 and Theano, where we train BNNs on MNIST, CIFAR-10 and SVHN, and achieve nearly state-of-the-art results. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which might lead to a great increase in power-efficiency. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available.

Motivation & Objective

• Motivate reducing memory usage and power consumption in neural networks by binarizing weights and activations.
• Investigate training-time gradient computation and run-time binarization to enable efficient learning.
• Evaluate BNNs on standard benchmarks (MNIST, CIFAR-10, SVHN) across different frameworks.
• Provide an optimized binary matrix multiplication kernel to accelerate inference/training.

Proposed method

• Represent weights and activations as binary values during forward pass and train-time gradient computation.
• Train using two frameworks (Torch7 and Theano) to validate framework-agnostic performance.
• Evaluate on MNIST, CIFAR-10, and SVHN to assess classification accuracy and efficiency.
• Develop a binary matrix multiplication GPU kernel to accelerate computations, demonstrated by a 7x speedup on MNIST.
• Make training and runtime code for BNNs publicly available.

Experimental results

Research questions

• RQ1 Can binarized neural networks achieve competitive accuracy on standard benchmarks (MNIST, CIFAR-10, SVHN) compared to full-precision models?
• RQ2 To what extent do binary weights/activations reduce memory footprint and arithmetic operations during forward and training passes?
• RQ3 How much speedup can be achieved with a binary matrix multiplication kernel for practical workloads (e.g., MNIST) without sacrificing accuracy?
• RQ4 Are the results consistent across different deep learning frameworks (Torch7, Theano)?

Key findings

• BNNs achieve nearly state-of-the-art results on MNIST, CIFAR-10, and SVHN.
• Forward passes in BNNs substantially reduce memory usage and rely on bit-wise operations to replace most arithmetic.
• A binary matrix multiplication GPU kernel yields a significant speedup (7x for MNIST) without accuracy loss.
• The authors provide training and runtime code for reproducibility and further research.