[Paper Review] XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Mohammad Rastegari, Vicente Ordóñez+2 more|arXiv (Cornell University)|Mar 16, 2016

Advanced Neural Network Applications36 references285 citations

TL;DR

The paper introduces Binary-Weight-Networks and XNOR-Networks that binarize weights (and inputs for XNOR) to drastically reduce memory and computation while preserving imageNet accuracy close to full-precision networks. XNOR-Nets enable CPU-based real-time inference on limited-resource devices.

ABSTRACT

We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks. In Binary-Weight-Networks, the filters are approximated with binary values resulting in 32x memory saving. In XNOR-Networks, both the filters and the input to convolutional layers are binary. XNOR-Networks approximate convolutions using primarily binary operations. This results in 58x faster convolutional operations and 32x memory savings. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time. Our binary networks are simple, accurate, efficient, and work on challenging visual tasks. We evaluate our approach on the ImageNet classification task. The classification accuracy with a Binary-Weight-Network version of AlexNet is only 2.9% less than the full-precision AlexNet (in top-1 measure). We compare our method with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than 16% in top-1 accuracy.

Motivation & Objective

• Motivate efficient CNNs for resource-constrained devices without huge accuracy loss on ImageNet.
• Propose two binarization schemes: Binary-Weight-Networks (weights are binary) and XNOR-Networks (weights and inputs binary).
• Demonstrate that binary approximations can achieve competitive ImageNet performance and substantial speedups/memory savings.

Proposed method

• Approximate convolutions with binary operations using a scaling factor; for Binary-Weight-Networks W ≈ αB with B ∈ {+1,−1}^n and α ∈ R^+.
• Derive optimal B as sign(W) and α as (1/n) ||W||_1.
• Train binary-weight networks by binarizing weights during forward/backward passes while keeping real-valued weights for updates.
• Extend to XNOR-Networks where both inputs and weights are binary; approximate dot products with XNOR and bitcounting, with per-subtensor scaling factors β and α.
• Compute β efficiently by convolving the input’s average absolute values with a normalized kernel to obtain scaling across all sub-tensors.
• Train XNOR-Networks with a binary activation/gradient framework and allow a non-binary activation after binary convolution to improve accuracy.

Experimental results

Research questions

• RQ1 Can CNNs trained with binary weights retain competitive accuracy on ImageNet?
• RQ2 Do fully binary networks (weights and inputs) provide substantial speedups on CPUs while maintaining accuracy?
• RQ3 What are the trade-offs in using binary scaling factors and binarization during training and inference?
• RQ4 How do Binary-Weight-Networks and XNOR-Networks compare to prior binarization methods on large-scale datasets?

Key findings

• Binary-Weight-Networks can match full-precision AlexNet accuracy on ImageNet in their experiments.
• XNOR-Networks significantly reduce memory and enable CPU-based real-time inference with substantial speedups (reported as up to ~58× in high-precision operation terms).
• XNOR-Nets outperform prior binary-method baselines (BinaryConnect and BinaryNet) by large margins on ImageNet top-1 accuracy (over 16% relative improvement cited).
• Including a scaling factor for weights (α) is more impactful than scaling inputs (β) for accuracy.
• Binary-gradient variant (using binary gradient in training) reduces top-1 accuracy by only about 1.4% in the reported settings.
• Experiments cover AlexNet, ResNet-18, and GoogLeNet variants, showing varying degradation when moving from full precision to binary representations.