[Paper Review] How Do Adam and Training Strategies Help BNNs Optimization?

Zechun Liu, Zhiqiang Shen+4 more|arXiv (Cornell University)|Jun 21, 2021

Advanced Neural Network Applications26 citations

TL;DR

The paper analyzes why Adam outperforms SGD for Binary Neural Networks (BNNs) and introduces a weight-decay–based training scheme that, on the same architecture, boosts ImageNet top-1 accuracy to 70.5% with a simple two-step training approach.

ABSTRACT

The best performing Binary Neural Networks (BNNs) are usually attained using Adam optimization and its multi-step training variants. However, to the best of our knowledge, few studies explore the fundamental reasons why Adam is superior to other optimizers like SGD for BNN optimization or provide analytical explanations that support specific training strategies. To address this, in this paper we first investigate the trajectories of gradients and weights in BNNs during the training process. We show the regularization effect of second-order momentum in Adam is crucial to revitalize the weights that are dead due to the activation saturation in BNNs. We find that Adam, through its adaptive learning rate strategy, is better equipped to handle the rugged loss surface of BNNs and reaches a better optimum with higher generalization ability. Furthermore, we inspect the intriguing role of the real-valued weights in binary networks, and reveal the effect of weight decay on the stability and sluggishness of BNN optimization. Through extensive experiments and analysis, we derive a simple training scheme, building on existing Adam-based optimization, which achieves 70.5% top-1 accuracy on the ImageNet dataset using the same architecture as the state-of-the-art ReActNet while achieving 1.1% higher accuracy. Code and models are available at https://github.com/liuzechun/AdamBNN.

Motivation & Objective

• Investigate why Adam yields better generalization for BNNs than SGD.
• Understand how activation saturation and second-mmoment normalization affect BNN optimization.
• Examine the role of real-valued latent weights and weight decay in BNN training stability.
• Propose practical training strategies that improve performance without changing architectures.

Proposed method

• Compare Adam and SGD on large-scale ImageNet training for BNNs using the same architectures.
• Visualize gradient and weight trajectories to explain optimization dynamics under binarization.
• Introduce metrics (FF ratio and C2I ratio) to quantify stability and initialization dependency.
• Analyze the effect of weight decay on real-valued latent weights and binary weight stability.
• Propose a two-step training scheme and weight-decay strategy to improve accuracy, validated against state-of-the-art BNNs.

Experimental results

Research questions

• RQ1 Why is Adam more effective than SGD for optimizing binary neural networks on ImageNet?
• RQ2 How do activation saturation and second-moment normalization influence gradient flow and weight updates in BNNs?
• RQ3 What is the impact of weight decay on the stability and initialization dependence of BNN optimization, and how can training be improved accordingly?

Key findings

• Adam achieves higher top-1 accuracy than SGD on BNNs across settings (e.g., 61.49% vs 58.98% in one experiment).
• Activation saturation leads to zero or vanishing gradients, which Adam mitigates via its adaptive learning rates and second-moment normalization.
• Real-valued latent weights act as confidence scores for binary signs; Adam yields heavier tails and more stable, well-trained weight distributions than SGD.
• Weight decay in real-valued latent weights affects stability and initialization dependence, with a carefully tuned two-step training scheme yielding 70.5% top-1 on ImageNet, surpassing prior BNNs.