[Paper Review] XNOR-Net++: Improved Binary Neural Networks

Adrian Bulat, Georgios Tzimiropoulos|arXiv (Cornell University)|Sep 30, 2019

Advanced Neural Network Applications42 references119 citations

TL;DR

XNOR-Net++ learns a single discriminative scaling factor for binary weights and activations, exploring factor shapes to boost accuracy; achieves up to 6% absolute gain on ImageNet with ResNet-18 within the same budget.

ABSTRACT

This paper proposes an improved training algorithm for binary neural networks in which both weights and activations are binary numbers. A key but fairly overlooked feature of the current state-of-the-art method of XNOR-Net is the use of analytically calculated real-valued scaling factors for re-weighting the output of binary convolutions. We argue that analytic calculation of these factors is sub-optimal. Instead, in this work, we make the following contributions: (a) we propose to fuse the activation and weight scaling factors into a single one that is learned discriminatively via backpropagation. (b) More importantly, we explore several ways of constructing the shape of the scale factors while keeping the computational budget fixed. (c) We empirically measure the accuracy of our approximations and show that they are significantly more accurate than the analytically calculated one. (d) We show that our approach significantly outperforms XNOR-Net within the same computational budget when tested on the challenging task of ImageNet classification, offering up to 6\% accuracy gain.

Motivation & Objective

• Motivate improved accuracy for fully binarized networks on large-scale datasets like ImageNet.
• Propose discriminatively learned, fused scaling factors for binary convolutions to replace analytic scaling.
• Explore various shapes of the scaling factor Gamma to balance expressivity and computational budget.
• Demonstrate that the method improves performance across shallow and deep architectures under the same budget.
• Quantify gains in accuracy and efficiency relative to prior binary network approaches.

Proposed method

• Fuse activation and weight scaling factors into a single Gamma learned via backpropagation.
• Propose four constructions for Gamma to vary expressivity while keeping test-time cost fixed (Cases 1–4).
• Show that discriminatively learned Gamma yields lower quantization error than analytic scaling.
• Evaluate on ImageNet using both AlexNet and ResNet-18, under a fully binarized regime.
• Provide efficiency analysis comparing theoretical speed-ups and storage with prior XNOR-Net.

Experimental results

Research questions

• RQ1 Can learning a single discriminative scaling factor Gamma outperform analytically computed scaling in XNOR-Net?
• RQ2 How do different Gamma shapes (Cases 1–4) affect accuracy and parameter efficiency?
• RQ3 Is the improved training approach agnostic to network architecture (AlexNet and ResNet-18) when binarized?
• RQ4 What is the practical speed-up and memory benefit of XNOR-Net++ under fixed budget compared to XNOR-Net?

Key findings

• XNOR-Net++ achieves up to 6% absolute Top-1 accuracy gain over the state-of-the-art XNOR-Net on ImageNet with ResNet-18.
• Learning a single Gamma outperforms analytic scaling by significantly reducing reconstruction error between binary and real-valued convolutions.
• Among Gamma shapes, Case 4 (alpha⊗beta⊗gamma) provides the best accuracy among the proposed constructions.
• Bi-binaraized networks with XNOR-Net++ outperform prior binarized networks on ImageNet, with competitive efficiency and substantial memory savings (≈64×).
• Empirical results show lower L1 reconstruction error for the proposed Gamma constructions compared to the analytic baseline.