[Paper Review] Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

Cong Leng, Hao Li+2 more|arXiv (Cornell University)|Jul 24, 2017

Sparse and Compressive Sensing Techniques79 citations

TL;DR

The paper formulates training extremely low-bit neural networks as a discretely constrained problem and solves it with ADMM, achieving strong results on ImageNet and Pascal VOC across several architectures.

ABSTRACT

Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.

Motivation & Objective

• Motivate the need to compress deep nets to extremely low bit-widths for memory and computation efficiency.
• Model the training of such networks as a discretely constrained optimization problem (mixed integer programming).
• Decouple continuous weights from discrete constraints using ADMM to enable tractable subproblems.
• Propose proximal (extragradient) and iterative quantization steps to solve subproblems efficiently.
• Demonstrate effectiveness on image classification (ImageNet) and object detection (Pascal VOC) across multiple architectures.

Proposed method

• Formulate the training as min_W f(W) subject to W in the constrained set C that encodes weights as zeros and powers of two with layer-wise scaling factors alpha_i.
• Decouple W and a discrete auxiliary variable G with W = G via ADMM, introducing the augmented Lagrangian and alternating updates.
• Solve the proximal step (continuous) using extragradient method to accelerate convergence.
• Solve the projection step (discrete) by iteratively solving for (Q_i, alpha_i) to minimize ||V_i - alpha_i Q_i||^2 with Q_i in {0, ±1, ±2, ..., ±2^N} and alpha_i > 0.
• Perform dual update to accumulate constraint violation through lambda (scaled dual variable).
• Apply iterative quantization to converge to a local optimum for the discrete projection per layer.

Experimental results

Research questions

• RQ1 Can ADMM effectively decouple discrete weight constraints from continuous optimization to learn extremely low-bit neural networks?
• RQ2 How does the proposed ADMM-based method compare to state-of-the-art low-bit approaches (e.g., Binary Weight Network, Ternary Weight Network) across standard CNN architectures?
• RQ3 What is the performance impact of extremely low-bit quantization on large-scale datasets (ImageNet) and object detection benchmarks (Pascal VOC) across various architectures?
• RQ4 How do per-layer scale factors alpha_i influence the optimization and final accuracy?

Key findings

• The method achieves lossless compression with only 3 bits on AlexNet and VGG-16 compared to full precision.
• On ResNet-18, binary quantization outperforms BWN by about 4 points in top-1 and 3.2 points in top-5 accuracy; ternary outperforms TWN by about 5.2 and 3.3 points respectively.
• For ResNet-50, binary and ternary quantization substantially surpass BWN and TWN with notable gains in top-1/top-5 accuracy.
• On GoogleNet, binary and ternary quantizations show larger degradation than ResNet but still outperform BWN and TWN; including INT8-adjusted 1x1 kernels improves results.
• Compared with TTN and INQ, the proposed ternary approach achieves competitive or superior top-1/top-5 accuracy on AlexNet and ResNet-18 (e.g., TTN: 0.575/0.797; Ours: 0.582/0.806 on AlexNet; TTN: 0.666/0.872; INQ: 0.660/0.871; Ours: 0.670/0.875 on ResNet-18).
• In object detection (Pascal VOC with SSD), ternary and low-bit quantization maintain competitive mAP compared to full precision, with minimal loss on VGG16+SSD and modest loss on Darknet+SSD.