[Paper Review] Learnable Companding Quantization for Accurate Low-bit Neural Networks

Kohei Yamamoto|arXiv (Cornell University)|Mar 12, 2021

Advanced Neural Network Applications28 references4 citations

TL;DR

This paper proposes Learnable Companding Quantization (LCQ), a novel non-uniform quantization method that jointly optimizes learnable companding functions and clipping parameters to minimize quantization error in 2-, 3-, and 4-bit neural networks. LCQ achieves state-of-the-art accuracy, with a 2-bit ResNet-50 on ImageNet reaching 75.1% top-1 accuracy, reducing the accuracy gap to just 1.7% compared to full-precision models.

ABSTRACT

Quantizing deep neural networks is an effective method for reducing memory consumption and improving inference speed, and is thus useful for implementation in resource-constrained devices. However, it is still hard for extremely low-bit models to achieve accuracy comparable with that of full-precision models. To address this issue, we propose learnable companding quantization (LCQ) as a novel non-uniform quantization method for 2-, 3-, and 4-bit models. LCQ jointly optimizes model weights and learnable companding functions that can flexibly and non-uniformly control the quantization levels of weights and activations. We also present a new weight normalization technique that allows more stable training for quantization. Experimental results show that LCQ outperforms conventional state-of-the-art methods and narrows the gap between quantized and full-precision models for image classification and object detection tasks. Notably, the 2-bit ResNet-50 model on ImageNet achieves top-1 accuracy of 75.1% and reduces the gap to 1.7%, allowing LCQ to further exploit the potential of non-uniform quantization.

Motivation & Objective

• To address the significant accuracy drop in extremely low-bit (2-, 3-, 4-bit) neural networks compared to full-precision models.
• To improve quantization accuracy by enabling flexible, non-uniform quantization level control through learnable companding functions.
• To stabilize training of quantized networks using a novel weight normalization technique.
• To reduce memory overhead for inference by optimizing lookup table (LUT) size without sacrificing performance.

Proposed method

• LCQ formulates a learnable piecewise linear companding function that nonlinearly compresses and expands weight and activation values to enable non-uniform quantization.
• The companding function is jointly optimized with the clipping threshold and network weights using backpropagation with a straight-through estimator (STE).
• A new normalization technique, Limited Weight Normalization (LWN), restores the standard deviation of quantized weights to improve training stability and accuracy.
• The method uses a re-quantization technique to reduce the outer bit-width of lookup tables (LUTs), minimizing memory cost for efficient inference.
• The companding function is parameterized with a set of learnable parameters Θ, allowing flexible and adaptive quantization level distribution based on data distribution.
• The approach integrates a composite function of clipping, compressing, rounding, and expanding, where the non-linear compressing and expanding components are jointly trained.

Experimental results

Research questions

• RQ1 Can learnable non-uniform quantization significantly reduce the accuracy gap between low-bit and full-precision models in image classification and object detection?
• RQ2 How does the number of intervals in the learnable companding function affect the performance of low-bit networks?
• RQ3 To what extent does Limited Weight Normalization (LWN) improve training stability and accuracy in quantized networks?
• RQ4 Can reducing the outer bit-width of lookup tables (LUTs) maintain high model accuracy while reducing memory overhead for inference?
• RQ5 How does LCQ compare to existing uniform and non-uniform quantization methods in terms of accuracy and efficiency across diverse architectures and datasets?

Key findings

• The 2-bit ResNet-50 model on ImageNet achieved a top-1 accuracy of 75.1%, reducing the accuracy gap to only 1.7% compared to the full-precision model.
• LCQ outperformed state-of-the-art methods on CIFAR-10/100, ImageNet, and COCO datasets for both image classification and object detection tasks.
• Increasing the number of intervals in the companding function improved accuracy, especially for lower bit-widths, indicating a strong link between flexibility and performance.
• Limited Weight Normalization (LWN) provided a notable accuracy improvement at 2-bit quantization and minor gains at 3- and 4-bit levels, enhancing training stability.
• Reducing the outer bit-width of LUTs from 8 to 4 bits maintained accuracy within 0.2% degradation on both CIFAR-10 and ImageNet, while halving the LUT memory cost.
• LCQ achieved comparable or better mean average precision (AP) than the non-uniform method APoT on COCO object detection, demonstrating strong generalization across tasks.