[Paper Review] LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

Dongqing Zhang, Jiaolong Yang+2 more|arXiv (Cornell University)|Jul 26, 2018

Advanced Neural Network Applications4 references45 citations

TL;DR

This paper proposes learnable quantizers jointly trained with networks to quantize weights and activations, achieving state-of-the-art accuracy for low-bit DNNs on CIFAR-10 and ImageNet by enabling bitwise-friendly, arbitrary-bit quantization.

ABSTRACT

Although weight and activation quantization is an effective approach for Deep Neural Network (DNN) compression and has a lot of potentials to increase inference speed leveraging bit-operations, there is still a noticeable gap in terms of prediction accuracy between the quantized model and the full-precision model. To address this gap, we propose to jointly train a quantized, bit-operation-compatible DNN and its associated quantizers, as opposed to using fixed, handcrafted quantization schemes such as uniform or logarithmic quantization. Our method for learning the quantizers applies to both network weights and activations with arbitrary-bit precision, and our quantizers are easy to train. The comprehensive experiments on CIFAR-10 and ImageNet datasets show that our method works consistently well for various network structures such as AlexNet, VGG-Net, GoogLeNet, ResNet, and DenseNet, surpassing previous quantization methods in terms of accuracy by an appreciable margin. Code available at https://github.com/Microsoft/LQ-Nets

Motivation & Objective

• Motivate reducing DNN model size and compute by quantizing weights and activations while minimizing accuracy loss.
• Propose learnable quantizers that are compatible with bitwise operations to preserve fast inference.
• Enable layer-wise/channel-wise quantization with arbitrary bit-widths and train quantizers jointly with network parameters.
• Demonstrate improved accuracy over prior quantization methods across multiple architectures and datasets.

Proposed method

• Introduce a learnable quantizer Qours(x, v) = v^T e_l for x in a quantization interval, with e_l encoding K-bit levels.
• Quantization levels are generated by a learned basis v ∈ R^K and K-bit encodings e_l ∈ {−1,1}^K, enabling bitwise-compatible quantization.
• Use layer-wise activation quantizers and channel-wise weight quantizers, with QEM (Quantization Error Minimization) to optimize v and encodings B alternately in forward passes.
• Forward pass optimization minimizes ||Bx − vB||^2 with B ∈ {−1,1}^{K×N} and v ∈ R^K; a closed-form update v* = (BB^T)^{-1}Bx is used when B is fixed.
• Backpropagate through quantizers with Straight-Through Estimator (STE); use moving averages to stabilize quantizer parameters during training.
• Train networks in the standard way, quantizers are discarded after training and only their codes/bases are kept for inference.

Experimental results

Research questions

• RQ1 Can learnable, bitwise-compatible quantizers improve accuracy of low-bit quantized networks beyond hand-crafted quantizers?
• RQ2 How does jointly training quantizers with network weights/activations affect performance across architectures and datasets?
• RQ3 What is the effectiveness of the Quantization Error Minimization (QEM) approach versus standard backpropagation for optimizing quantizers?
• RQ4 Does layer-wise activation quantization and channel-wise weight quantization with arbitrary bit-widths maintain inference speed while reducing model size?

Key findings

• Learnable quantizers consistently outperform fixed quantizers across architectures (AlexNet, VGG, GoogLeNet, ResNet, DenseNet).
• QEM-based training yields better results than standard backprop for both weight-only and weight-activation quantization.
• On ImageNet ResNet-18, 4/32 bits achieves top-1 70.0% and top-5 89.1%, very close to full precision (69.6%/89.2%); 2/32 and 3/32 also show substantial gains over fixed quantizers.
• On CIFAR-10, 3/32 quantization nearly matches full-precision results for ResNet-20 and VGG-Small; 1-bit weights with 2-bit activations still retain competitive accuracy.
• Compared with DoReFa-Net and HWGQ, LQ-Nets with 1/2, 2/2, 3/3, and 4/4 configurations achieve higher Top-1/Top-5 on ImageNet across ResNet-34, ResNet-50, and DenseNet-121.
• Training time increases modestly with bit-width, with 2/32, 3/32, and 1/2 showing proportional training time increases relative to full-precision.