[Paper Review] Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization

Koen Helwegen, James Widdicombe+4 more|arXiv (Cornell University)|Jun 5, 2019

Advanced Neural Network ApplicationsComputer Science35 references38 citations

TL;DR

The paper reframes latent real-valued weights in Binarized Neural Networks (BNNs) as inertia rather than true weights, introduces a latent-free optimizer (Bop) based on gradient consistency, and demonstrates competitive results on CIFAR-10 and ImageNet.

ABSTRACT

Optimization of Binarized Neural Networks (BNNs) currently relies on real-valued latent weights to accumulate small update steps. In this paper, we argue that these latent weights cannot be treated analogously to weights in real-valued networks. Instead their main role is to provide inertia during training. We interpret current methods in terms of inertia and provide novel insights into the optimization of BNNs. We subsequently introduce the first optimizer specifically designed for BNNs, Binary Optimizer (Bop), and demonstrate its performance on CIFAR-10 and ImageNet. Together, the redefinition of latent weights as inertia and the introduction of Bop enable a better understanding of BNN optimization and open up the way for further improvements in training methodologies for BNNs. Code is available at: https://github.com/plumerai/rethinking-bnn-optimization

Motivation & Objective

• Reassess the role of latent weights in deterministic BNN training.
• Propose a new optimizer tailored for BNNs that forgoes latent weights.
• Empirically evaluate the proposed approach on CIFAR-10 and ImageNet.
• Provide insights that guide future BNN optimization beyond real-valued approximations.

Proposed method

• Represent latent weights as w̃ = sign(w̃) · |w̃| to separate binarized weight and inertia.
• Interpret inertia magnitude as the key to when a binary weight flips, rather than as a true weight parameter.
• Propose Binary Optimizer (Bop) that flips weights based on an exponential moving average of gradients and a threshold, eliminating latent weights during optimization.
• Use an exponential moving average m_t = (1−γ)m_{t−1} + γ g_t and flip rule: flip if |m_t^i| ≥ τ and sign(m_t^i) = sign(w^{i}_{t−1}).
• Clip or scale latent magnitudes is replaced by γ (adaptivity) and τ (threshold) to control consistency and strength of signals.
• Demonstrate Bop’s properties and compare to latent-weight baselines across CIFAR-10 and ImageNet.

Experimental results

Research questions

• RQ1 Do latent real-valued weights function mainly as inertia rather than true optimization variables in BNNs?
• RQ2 Can a latent-free optimizer designed around gradient consistency (Bop) match or exceed latent-weight based training for BNNs?
• RQ3 What are the empirical benefits of Bop on standard benchmarks like CIFAR-10 and ImageNet?
• RQ4 How do hyperparameters γ (adaptivity rate) and τ (threshold) influence learning dynamics and generalization in BNN optimization?

Key findings

• Latent weights are better understood as encoding inertia; changing their magnitude does not change forward behavior unless a sign flip occurs.
• Theorem 1 shows learning-rate scaling invariance under certain conditions, implying per-weight learning rates can be absorbed into initialization.
• Bop, a latent-free optimizer based on gradient consistency and a flip threshold, achieves competitive CIFAR-10 results (40–41% top-1 with latent baselines around 40–41%) and improved stability.
• On ImageNet, Bop delivers competitive Top-1 and Top-5 accuracies across BinaryNet, XNOR-Net, and BiReal-Net compared to latent-weight baselines (e.g., BinaryNet: 41.1% vs 40.1%; XNOR-Net: 45.9% vs 44.2%; BiReal-Net: 56.6% vs 56.4%).
• Bop reduces memory requirements during training (one real-valued variable per weight) and relies on only two intuitive hyperparameters (γ and τ).
• The inertia-focused view provides a pathway to further improvements in regularization and knowledge distillation for BNNs.