[Paper Review] Regularizing Activation Distribution for Training Binarized Deep Networks

Ruizhou Ding, Ting-Wu Chin+2 more|arXiv (Cornell University)|Apr 4, 2019

Advanced Neural Network Applications44 references20 citations

TL;DR

This paper proposes a distribution loss to regularize activation distributions in binarized neural networks (BNNs), improving training stability and accuracy without sacrificing energy efficiency. By explicitly penalizing degenerate, saturated, or poorly activated channels, the method enables robust training across diverse optimizers and hyper-parameters, achieving state-of-the-art accuracy on ImageNet with 1-bit weights and activations.

ABSTRACT

Binarized Neural Networks (BNNs) can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN's energy efficiency. In this paper, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.

Motivation & Objective

• Address the difficulty of training BNNs with 1-bit weights and activations due to activation degeneration, saturation, and gradient mismatch.
• Overcome the limitations of prior approaches that increase network width or use floating-point scaling factors, which reduce energy efficiency.
• Develop a differentiable, learnable regularization mechanism that explicitly shapes activation distributions to avoid training instabilities.
• Demonstrate that the proposed method improves BNN accuracy while maintaining pure-logical computation and minimal memory footprint.

Proposed method

• Formulate a distribution loss $ L_D $ that penalizes activation distributions deviating from a desired, well-behaved shape (e.g., centered at zero with balanced positive/negative ratios).
• Introduce a differentiable approximation of the activation distribution using a smooth histogram-based estimation, enabling backpropagation through the loss.
• Combine the distribution loss with the standard classification loss via a hyper-parameter $ \\-lambda \$ to balance regularization and accuracy.
• Apply the loss during backpropagation to encourage per-channel activations to avoid degeneration (constant output), saturation (zero gradients), and gradient mismatch (poor STE approximation).
• Use a pre-activation batch normalization and ReLU-based structure to stabilize training, with the distribution loss acting as a regularizer on the activation flow.
• Enable end-to-end training by ensuring the entire loss function is differentiable, allowing gradients to flow through the distribution loss to network weights.

Experimental results

Research questions

• RQ1 Can explicit regularization of activation distributions improve the training stability and accuracy of BNNs with 1-bit weights and activations?
• RQ2 Does the proposed distribution loss reduce degeneration, saturation, and gradient mismatch in binarized networks?
• RQ3 Can the method improve BNN accuracy without introducing floating-point operations or increasing network width?
• RQ4 How robust is the distribution loss to variations in optimizer, learning rate, and other hyper-parameters?
• RQ5 Does the regularization effect hold across different network architectures, including VGG-style and residual networks?

Key findings

• The proposed distribution loss significantly improves BNN accuracy on CIFAR-10, achieving 90.12% top-1 accuracy with $ \\-lambda = 0.2 $, compared to 87.39% for the baseline BNN.
• On ImageNet, the method achieves state-of-the-art accuracy of 44.1% top-1 accuracy with 1-bit weights and activations, outperforming prior SOTA approaches.
• The distribution loss reduces activation degeneration: histograms show fewer channels with constant outputs and more balanced positive ratios.
• BNNs trained with the distribution loss are robust to optimizer choice—Adam, SGD with momentum, Nesterov, and RMSprop all yield consistent improvements.
• The method remains effective across a wide range of $ \\-lambda \$ values (0.2 to 2000), with accuracy consistently above baseline, indicating low sensitivity to regularization strength.
• The distribution loss decays quickly in early training, indicating rapid stabilization of activation distributions, which correlates with faster convergence and better generalization.