Skip to main content
QUICK REVIEW

[Paper Review] Binarized Neural Networks

Itay Hubara, Daniel Soudry|arXiv (Cornell University)|Feb 8, 2016
Advanced Neural Network ApplicationsComputer Science919 citations
TL;DR

The paper trains Binarized Neural Networks with binary weights and activations, achieving near state-of-the-art results on MNIST, CIFAR-10, and SVHN, while greatly reducing memory/computation and enabling a 7x speedup for MNIST with a binary GEMM kernel.

ABSTRACT

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time and when computing the parameters' gradient at train-time. We conduct two sets of experiments, each based on a different framework, namely Torch7 and Theano, where we train BNNs on MNIST, CIFAR-10 and SVHN, and achieve nearly state-of-the-art results. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which might lead to a great increase in power-efficiency. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available.

Motivation & Objective

  • Motivate reducing memory usage and power consumption in neural networks by binarizing weights and activations.
  • Investigate training-time gradient computation and run-time binarization to enable efficient learning.
  • Evaluate BNNs on standard benchmarks (MNIST, CIFAR-10, SVHN) across different frameworks.
  • Provide an optimized binary matrix multiplication kernel to accelerate inference/training.

Proposed method

  • Represent weights and activations as binary values during forward pass and train-time gradient computation.
  • Train using two frameworks (Torch7 and Theano) to validate framework-agnostic performance.
  • Evaluate on MNIST, CIFAR-10, and SVHN to assess classification accuracy and efficiency.
  • Develop a binary matrix multiplication GPU kernel to accelerate computations, demonstrated by a 7x speedup on MNIST.
  • Make training and runtime code for BNNs publicly available.

Experimental results

Research questions

  • RQ1Can binarized neural networks achieve competitive accuracy on standard benchmarks (MNIST, CIFAR-10, SVHN) compared to full-precision models?
  • RQ2To what extent do binary weights/activations reduce memory footprint and arithmetic operations during forward and training passes?
  • RQ3How much speedup can be achieved with a binary matrix multiplication kernel for practical workloads (e.g., MNIST) without sacrificing accuracy?
  • RQ4Are the results consistent across different deep learning frameworks (Torch7, Theano)?

Key findings

  • BNNs achieve nearly state-of-the-art results on MNIST, CIFAR-10, and SVHN.
  • Forward passes in BNNs substantially reduce memory usage and rely on bit-wise operations to replace most arithmetic.
  • A binary matrix multiplication GPU kernel yields a significant speedup (7x for MNIST) without accuracy loss.
  • The authors provide training and runtime code for reproducibility and further research.

Read this paper

Open the full text in the Nubint reader and ask about any passage as you read.

Read this paper on Nubint

This review was generated by AI from the original paper. Please check the paper before citing it.