[Paper Review] Bitwise Neural Networks

Minje Kim, Paris Smaragdis|arXiv (Cornell University)|Jan 22, 2016

Neural Networks and Applications19 references174 citations

TL;DR

This paper introduces Bitwise Neural Networks (BNNs) where all inputs, weights, biases, and activations are binary, enabling feedforward via XNOR and bit counting, with training including weight compression and noisy backpropagation; tested on MNIST with binary features showing competitive accuracy and substantial computational savings.

ABSTRACT

Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.

Motivation & Objective

• Motivate the need for highly resource-efficient neural networks for embedded and on-device applications.
• Propose a fully binary neural network where all components operate with bits using XNOR and bit counting.
• Develop training strategies: real-valued weight compression precedes binarized training with noisy backpropagation.
• Demonstrate that BNNs can achieve competitive performance on MNIST with significantly reduced computational requirements.

Proposed method

• Represent all weights, biases, inputs, and hidden outputs as bipolar binaries; feedforward uses XNOR and bit counting instead of multiplication/addition.
• Use a sign activation to maintain bipolar binary signals across layers.
• First train a real-valued network with weight compression to ease conversion to BNNs.
• Train the actual BNN with binarized weights/signals using noisy backpropagation and update corresponding real-valued proxies during learning.
• Binaraize inputs and reinitialize/binarize weights after each update; use a softmax output layer for multiclass MNIST evaluation.

Experimental results

Research questions

• RQ1 Can a fully binary (bitwise) neural network match the performance of a real-valued network on image recognition tasks when trained with weight compression and noisy backpropagation?
• RQ2 What is the trade-off in accuracy between bitwise inputs/weights and their real-valued counterparts on MNIST with binary feature representations?
• RQ3 Do bitwise operations (XNOR and bit counting) offer substantial computational and memory savings while preserving accuracy?

Key findings

• BNNs achieve competitive classification performance with small accuracy loss compared to real-valued networks on MNIST with binary features.
• Baseline real-valued networks (64-bit floating point) achieve error around 1.17% (bipolar input), 1.32% (0/1 input), and 1.36% (fixed-point 2-bit input).
• BNNs trained with binarized inputs yield test errors of approximately 1.33% (bipolar), 1.36% (0/1), and 1.47% (fixed-point 2-bit).
• The bitwise approach provides dramatic computational savings due to replacing multiplications/additions with simple bitwise operations, with only modest performance degradation.
• The study demonstrates that fully binary networks can operate effectively on standard recognition tasks when aided by weight compression and noisy backpropagation.