[Paper Review] Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

Aojun Zhou, Anbang Yao+3 more|arXiv (Cornell University)|Feb 9, 2017

Advanced Neural Network ApplicationsComputer Science591 citations

TL;DR

This paper introduces Incremental Network Quantization (INQ), which converts any pre-trained full-precision CNN into a low-precision model where weights are powers of two or zero, using weight partition, group-wise quantization, and retraining in an iterative, lossless manner. It achieves or improves accuracy at 5-, 4-, and even 3-bit quantization on ImageNet across multiple architectures.

ABSTRACT

This paper presents incremental network quantization (INQ), a novel method, targeting to efficiently convert any pre-trained full-precision convolutional neural network (CNN) model into a low-precision version whose weights are constrained to be either powers of two or zero. Unlike existing methods which are struggled in noticeable accuracy loss, our INQ has the potential to resolve this issue, as benefiting from two innovations. On one hand, we introduce three interdependent operations, namely weight partition, group-wise quantization and re-training. A well-proven measure is employed to divide the weights in each layer of a pre-trained CNN model into two disjoint groups. The weights in the first group are responsible to form a low-precision base, thus they are quantized by a variable-length encoding method. The weights in the other group are responsible to compensate for the accuracy loss from the quantization, thus they are the ones to be re-trained. On the other hand, these three operations are repeated on the latest re-trained group in an iterative manner until all the weights are converted into low-precision ones, acting as an incremental network quantization and accuracy enhancement procedure. Extensive experiments on the ImageNet classification task using almost all known deep CNN architectures including AlexNet, VGG-16, GoogleNet and ResNets well testify the efficacy of the proposed method. Specifically, at 5-bit quantization, our models have improved accuracy than the 32-bit floating-point references. Taking ResNet-18 as an example, we further show that our quantized models with 4-bit, 3-bit and 2-bit ternary weights have improved or very similar accuracy against its 32-bit floating-point baseline. Besides, impressive results with the combination of network pruning and INQ are also reported. The code is available at https://github.com/Zhouaojun/Incremental-Network-Quantization.

Motivation & Objective

• Motivate and address accuracy loss and long convergence in low-precision CNN quantization.
• Propose a lossless incremental quantization framework for converting full-precision CNNs to low-precision weights.
• Demonstrate effectiveness across major architectures on ImageNet.
• Explore the benefits of combining INQ with network pruning for compression.
• Show practical bit-width limits and convergence behavior of INQ.

Proposed method

• Introduce weight partition to split weights into a low-precision base and a retrainable compensation group.
• Apply group-wise quantization with a variable-length encoding to quantize the base weights to powers of two or zero.
• Perform retraining on the compensation group to recover accuracy while keeping base weights fixed.
• Iterate the three operations (partition, quantization, retraining) until all weights are quantized.
• Use a constrained optimization: minimize L(W) + λR(W) subject to W(i,j) ∈ P_l for the quantized group, with SGD updates that only affect non-quantized weights.
• Equation references include: weight quantization rule (4) for mapping to P_l, n1/n2 determination (2,3), and the masked SGD update (8).

Experimental results

Research questions

• RQ1 Can full-precision CNNs be quantized to low-precision weights without accuracy loss using INQ?
• RQ2 How does weight partition strategy affect final accuracy and convergence?
• RQ3 What are the attainable bit-widths that permit lossless or near-lossless quantization on large-scale datasets?
• RQ4 How does INQ interact with pruning and other compression techniques for CNNs on ImageNet?

Key findings

• 5-bit INQ across AlexNet, VGG-16, GoogleNet, ResNet-18 and ResNet-50 yields consistent top-1/top-5 improvements over the corresponding full-precision baselines (top-1 gains 0.13%–2.28%, top-5 gains 0.23%–1.65%).
• INQ shows easy convergence, typically converging with fewer than 8 retraining epochs per iteration to achieve lossless 5-bit quantization.
• ResNet-18 with 4-bit, 3-bit, and 2-bit ternary weights achieves comparable or improved accuracy relative to the 32-bit baseline (4-bit and 3-bit are very close; 2-bit ternary is worse than baseline but better than some prior binary/ternary models).
• Pruning+INQ outperforms the deep compression method of Han et al. (2016) for AlexNet, achieving higher compression with maintained or improved accuracy (e.g., 53x for 5-bit INQ+DNS vs. 27x/35x in the prior work).
• When compared to vector quantization alone, INQ demonstrates superior accuracy retention (5-bit/4-bit quantization) and quantizes all layers rather than just fully connected layers.
• INQ can achieve substantial compression while maintaining or improving accuracy, enabling practical deployment on resource-constrained devices.