[Paper Review] AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets

Zhijun Tu, Xinghao Chen+2 more|arXiv (Cornell University)|Aug 17, 2022

Advanced Neural Network Applications4 citations

TL;DR

AdaBin proposes a novel adaptive binary quantization method for Binary Neural Networks (BNNs) that learns optimal real-valued binary sets \{b₁, b₂\} for weights and activations instead of fixed \{-1, +1\}, using statistical equalization for weights and gradient-based optimization for activations. This approach achieves state-of-the-art performance, attaining 66.4% Top-1 accuracy on ImageNet with ResNet-18 and 69.4 mAP on PASCAL VOC with SSD300.

ABSTRACT

This paper studies the Binary Neural Networks (BNNs) in which weights and activations are both binarized into 1-bit values, thus greatly reducing the memory usage and computational complexity. Since the modern deep neural networks are of sophisticated design with complex architecture for the accuracy reason, the diversity on distributions of weights and activations is very high. Therefore, the conventional sign function cannot be well used for effectively binarizing full-precision values in BNNs. To this end, we present a simple yet effective approach called AdaBin to adaptively obtain the optimal binary sets $\{b_1, b_2\}$ ($b_1, b_2\in \mathbb{R}$) of weights and activations for each layer instead of a fixed set ( extit{i.e.}, $\{-1, +1\}$). In this way, the proposed method can better fit different distributions and increase the representation ability of binarized features. In practice, we use the center position and distance of 1-bit values to define a new binary quantization function. For the weights, we propose an equalization method to align the symmetrical center of binary distribution to real-valued distribution, and minimize the Kullback-Leibler divergence of them. Meanwhile, we introduce a gradient-based optimization method to get these two parameters for activations, which are jointly trained in an end-to-end manner. Experimental results on benchmark models and datasets demonstrate that the proposed AdaBin is able to achieve state-of-the-art performance. For instance, we obtain a 66.4% Top-1 accuracy on the ImageNet using ResNet-18 architecture, and a 69.4 mAP on PASCAL VOC using SSD300. The PyTorch code is available at \url{https://github.com/huawei-noah/Efficient-Computing/tree/master/BinaryNetworks/AdaBin} and the MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/AdaBin}.

Motivation & Objective

• Address the severe accuracy degradation in BNNs caused by fixed binary sets \{-1, +1\} that poorly match diverse weight and activation distributions.
• Overcome the limitation of the sign function in binarizing features with complex, non-symmetric distributions across layers.
• Enhance the representational capacity of BNNs by learning optimal real-valued binary sets \{b₁, b₂\} that better match full-precision distributions.
• Enable end-to-end training of adaptive binary sets for both weights and activations while preserving hardware efficiency via XNOR and BitCount operations.
• Close the performance gap between BNNs and full-precision networks on complex vision tasks like object detection and segmentation.

Proposed method

• Define a new binary quantization function using center position and distance of 1-bit values, replacing the fixed sign function.
• For weights, apply an equalization method that aligns the symmetric center of the binary distribution to the real-valued distribution and minimizes Kullback-Leibler divergence (KLD) to derive analytic solutions for center and distance.
• For activations, introduce a gradient-based optimization method with learnable parameters for center and distance, initialized via the sign function and jointly trained end-to-end.
• Integrate a Maxout-like non-linearity with learnable scale factors \gamma⁺ and \gamma⁻ to improve activation representation without increasing bit-width.
• Maintain hardware efficiency by ensuring the final binary operations remain XNOR and BitCount, enabling ~64× speedup and ~32× memory reduction.
• Use KLD minimization and statistical alignment to ensure the learned binary sets preserve the statistical properties of the original full-precision tensors.

Experimental results

Research questions

• RQ1 Can learning adaptive binary sets \{b₁, b₂\} instead of fixed \{-1, +1\} improve the representational capacity of BNNs?
• RQ2 How does adaptive center and distance learning for weights and activations affect the performance of BNNs on complex datasets like ImageNet and PASCAL VOC?
• RQ3 To what extent can AdaBin close the accuracy gap between BNNs and full-precision models in object detection and classification tasks?
• RQ4 Does the combination of KLD minimization and equalization for weights, along with gradient-based optimization for activations, lead to consistent performance gains across different network architectures?
• RQ5 Can the proposed method maintain hardware efficiency (XNOR and BitCount) while significantly improving accuracy over prior BNNs?

Key findings

• AdaBin achieves 66.4% Top-1 accuracy on ImageNet using ResNet-18, outperforming previous SOTA BNNs by a significant margin.
• On PASCAL VOC with SSD300, AdaBin achieves 69.4 mAP, surpassing XNOR-Net by 13.8 mAP and Bi-Real Net by 5.6 mAP.
• The ablation study shows that combining AdaBin’s weight equalization and activation optimization boosts accuracy by 2% over vanilla BNNs on CIFAR-10.
• The Maxout activation with learnable \gamma⁺ and \gamma⁻ improves accuracy by 9.6% and 9.7% respectively, with both coefficients yielding the best result (88.2% on CIFAR-10).
• AdaBin with a shortcut structure outperforms 4-bit quantization methods like TWN and DoReFa, demonstrating that adaptive binarization can surpass multi-bit baselines.
• The method maintains theoretical acceleration of 60.85× and memory saving of 31×, confirming compatibility with efficient hardware deployment.