[Paper Review] Back to Simplicity: How to Train Accurate BNNs from Scratch?

Joseph Bethge, Haojin Yang+2 more|arXiv (Cornell University)|Jun 19, 2019

Advanced Neural Network Applications29 references44 citations

TL;DR

The paper demonstrates that highly accurate binary neural networks can be trained from scratch using simple training strategies and network designs, introducing BinaryDenseNet which surpasses prior 1-bit CNNs on ImageNet without extra tricks.

ABSTRACT

Binary Neural Networks (BNNs) show promising progress in reducing computational and memory costs but suffer from substantial accuracy degradation compared to their real-valued counterparts on large-scale datasets, e.g., ImageNet. Previous work mainly focused on reducing quantization errors of weights and activations, whereby a series of approximation methods and sophisticated training tricks have been proposed. In this work, we make several observations that challenge conventional wisdom. We revisit some commonly used techniques, such as scaling factors and custom gradients, and show that these methods are not crucial in training well-performing BNNs. On the contrary, we suggest several design principles for BNNs based on the insights learned and demonstrate that highly accurate BNNs can be trained from scratch with a simple training strategy. We propose a new BNN architecture BinaryDenseNet, which significantly surpasses all existing 1-bit CNNs on ImageNet without tricks. In our experiments, BinaryDenseNet achieves 18.6% and 7.6% relative improvement over the well-known XNOR-Network and the current state-of-the-art Bi-Real Net in terms of top-1 accuracy on ImageNet, respectively.

Motivation & Objective

• Question conventional wisdom on training BNNs and whether common tricks are necessary.
• Identify general design principles that preserve information flow in binary networks.
• Propose and validate a new BNN architecture (BinaryDenseNet) that reaches state-of-the-art accuracy.
• Demonstrate from-scratch training efficacy on ImageNet and CIFAR-10, with open-source code for reproducibility.

Proposed method

• Revisit common BNN techniques (scaling factors, approxsign, full-precision pre-training) and empirically assess their impact when training from scratch.
• Propose guidelines to maximize information flow, including encouraging shortcut connections and avoiding bottlenecks.
• Develop BinaryDenseNet by adapting DenseNet concepts to binary networks and evaluate downsampling strategies (full-precision vs binary).
• Compare with existing 1-bit CNNs on ImageNet and conduct an object detection preliminary study using SSD.
• Provide open-source BMXNet-based implementations to enable reproducibility.

Experimental results

Research questions

• RQ1 Can highly accurate BNNs be trained from scratch using standard training strategies without scaling tricks or pretraining?
• RQ2 What network design principles best preserve information flow in binary networks to mitigate precision loss?
• RQ3 Does increasing shortcut connections and avoiding bottlenecks yield measurable gains for BNNs on large-scale datasets?
• RQ4 How does BinaryDenseNet perform relative to state-of-the-art 1-bit CNNs (e.g., XNOR-Net, Bi-Real Net) on ImageNet?

Key findings

• BinaryDenseNet achieves state-of-the-art accuracy among 1-bit CNNs on ImageNet, surpassing prior methods by a notable margin.
• BinaryResNetE18 trained from scratch with the proposed principles outperforms several existing BNNs using the same architecture, with scaling factors not needed.
• On ImageNet, BinaryDenseNet models reach top-1 accuracies such as 60.7% (BinaryDenseNet28) and higher in larger variants, while Bi-Real Net and XNOR-Net are substantially lower in comparison.
• Full-precision downsampling layers can significantly improve accuracy on ImageNet (about 3% for binary ResNetE18) at the cost of larger model size.
• Increasing the number of shortcut connections (as in DenseNet-inspired designs) improves information flow and accuracy for BinaryDenseNet, with benefits observed when splitting blocks and enlarging connectivity.
• Their results show that training from scratch with the proposed design principles can outperform prior 1-bit CNNs, even without tricks like scaling, approxsign, or FP pre-training.