[Paper Review] Loss-aware Binarization of Deep Networks

Lu Hou, Quanming Yao+1 more|arXiv (Cornell University)|Nov 5, 2016

Face and Expression Recognition112 citations

TL;DR

The paper proposes Loss-Aware Binarization (LAB), a proximal Newton-based method with diagonal Hessian approximation to binarize neural network weights while directly minimizing the loss, improving over prior binarization methods especially for deep and recurrent nets.

ABSTRACT

Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time. Recently, there have been a number of attempts on binarizing the network weights and activations. This greatly reduces the network size, and replaces the underlying multiplications to additions or even XNOR bit operations. However, existing binarization schemes are based on simple matrix approximation and ignore the effect of binarization on the loss. In this paper, we propose a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss w.r.t. the binarized weights. The underlying proximal step has an efficient closed-form solution, and the second-order information can be efficiently obtained from the second moments already computed by the Adam optimizer. Experiments on both feedforward and recurrent networks show that the proposed loss-aware binarization algorithm outperforms existing binarization schemes, and is also more robust for wide and deep networks.

Motivation & Objective

• Motivate reducing neural network size and compute by binarizing weights and possibly activations.
• Directly minimize the loss with respect to binarized weights instead of just approximating full-precision weights.
• Leverage second-order information via a diagonal Hessian within a proximal Newton framework.
• Demonstrate robustness of LAB on feedforward and recurrent networks across several datasets.

Proposed method

• Formulate binarization as min_ŵ ℓ(ŵ) subject to ŵ_l = α_l b_l with α_l > 0 and b_l ∈ {±1}^{n_l}.
• Apply proximal Newton update with a diagonal Hessian D, using a second-order expansion of the loss and a proximal step.
• Derive a closed-form solution for the proximal step: α_l^t = ||D_l^{t-1} ⊙ w_l^t||_1 / ||D_l^{t-1}||_1 and b_l^t = sign(w_l^t) (Proposition 3.1).
• Use Adam-based estimates to form the diagonal Hessian surrogate D, leveraging second moments to precondition updates.
• Keep full-precision weights for updates while binarizing during forward/backward passes, enabling efficient training.
• Extend LAB to recurrent nets by binarizing input-to-hidden and hidden-to-hidden weights and addressing exploding gradients via curvature-aware scaling.

Experimental results

Research questions

• RQ1 How can binarization be performed so that the loss is minimized directly, rather than approximating full-precision weights?
• RQ2 Can a diagonal Hessian-based proximal Newton method provide a more effective binarization than existing schemes (BinaryConnect, Binary-Weight-Network) especially for deep or recurrent architectures?
• RQ3 Does incorporating second-order information improve stability and performance when binarizing weights in feedforward and recurrent networks?
• RQ4 What is the impact of LAB on standard benchmarks (MNIST, CIFAR-10, SVHN) and on language-modeling with LSTMs?

Key findings

• LAB outperforms existing binarization methods on MNIST, CIFAR-10, and SVHN in feedforward nets.
• LAB achieves test errors of 1.180% (MNIST), 10.500% (CIFAR-10), and 2.354% (SVHN), outperforming BinaryConnect and BWN and approaching or surpassing full-precision in some cases.
• LAB2 (weights+activations binarized) outperforms BNN and XNOR-Net variants in several settings.
• LAB demonstrates robustness for wide and deep networks and mitigates exploding gradient issues in recurrent nets where BinaryConnect struggles.