[Paper Review] AdaBits: Neural Network Quantization with Adaptive Bit-Widths

Qing Jin, Linjie Yang+1 more|arXiv (Cornell University)|Dec 20, 2019

Advanced Neural Network Applications50 references17 citations

TL;DR

AdaBits proposes a joint training approach to enable neural network quantization with adaptive bit-widths for weights and activations, allowing instant model adaptation to varying hardware constraints. By introducing Switchable Clipping Level (S-CL) to decouple clipping parameters per bit-width, it achieves accuracy comparable to individually quantized models across MobileNet V1/V2 and ResNet50 on ImageNet, with minimal performance drop even at low bit-widths like 2-bit and 3-bit.

ABSTRACT

Deep neural networks with adaptive configurations have gained increasing attention due to the instant and flexible deployment of these models on platforms with different resource budgets. In this paper, we investigate a novel option to achieve this goal by enabling adaptive bit-widths of weights and activations in the model. We first examine the benefits and challenges of training quantized model with adaptive bit-widths, and then experiment with several approaches including direct adaptation, progressive training and joint training. We discover that joint training is able to produce comparable performance on the adaptive model as individual models. We further propose a new technique named Switchable Clipping Level (S-CL) to further improve quantized models at the lowest bit-width. With our proposed techniques applied on a bunch of models including MobileNet-V1/V2 and ResNet-50, we demonstrate that bit-width of weights and activations is a new option for adaptively executable deep neural networks, offering a distinct opportunity for improved accuracy-efficiency trade-off as well as instant adaptation according to the platform constraints in real-world applications.

Motivation & Objective

• To enable real-time, hardware-adaptive deployment of deep neural networks by dynamically adjusting bit-widths of weights and activations during inference.
• To address the challenge of maintaining high model accuracy when quantizing to different bit-widths without retraining, especially at low bit-widths.
• To explore whether joint training with shared parameters across multiple bit-widths can match the performance of individually trained quantized models.
• To develop a mechanism that prevents interference between different bit-width configurations during training, particularly in clipping level optimization.
• To demonstrate that bit-width is a viable and effective degree of freedom for model compression and adaptive deployment beyond width, depth, and kernel size.

Proposed method

• Proposes a joint training framework where a single model is trained to support multiple bit-widths simultaneously, sharing all other parameters except for bit-width-specific clipping levels.
• Introduces Switchable Clipping Level (S-CL), a mechanism that assigns independent clipping level parameters per bit-width, preventing cross-bit-width optimization interference.
• Adapts the Scale-Adjusted Training (SAT) method as the baseline quantization scheme, integrating it into the joint training process for consistent quantization-aware training.
• Employs a differentiable routing mechanism to switch between bit-widths during training, enabling end-to-end optimization of all bit-width configurations.
• Uses a modified scheme that stores only full-precision weights and quantizes them on-the-fly per bit-width, reducing model size compared to storing quantized weights for each bit-width.
• Applies progressive training strategies as baselines for comparison, evaluating performance degradation when training in ascending or descending bit-width order.

Experimental results

Research questions

• RQ1 Can a single neural network be jointly trained to support multiple bit-widths for weights and activations while maintaining high accuracy across all configurations?
• RQ2 Does progressive training across bit-widths lead to performance degradation when fine-tuning from higher to lower bit-widths?
• RQ3 Can a shared model with bit-width-specific clipping levels outperform individually trained models at low bit-widths like 2-bit and 3-bit?
• RQ4 How does the Switchable Clipping Level (S-CL) mechanism improve quantization performance by decoupling clipping parameters per bit-width?
• RQ5 To what extent can adaptive bit-widths improve the accuracy-efficiency trade-off compared to traditional model compression techniques like width-multiplier scaling?

Key findings

• Joint training with S-CL achieves test accuracy on ImageNet that is nearly identical to individually quantized models across all bit-widths for MobileNet V1, MobileNet V2, and ResNet50.
• On 2-bit ResNet50, AdaBits with S-CL achieves a top-1 accuracy of 34.1%, significantly outperforming progressive training with descending bit-widths, which drops to 28.5%.
• For 4-bit and 3-bit ResNet50, AdaBits improves accuracy by 2.2% compared to progressive quantization with descending bit-widths, demonstrating strong generalization.
• The modified AdaBits scheme with S-CL increases accuracy by 0.3% on 4-bit MobileNet V1 compared to vanilla AdaBits, showing incremental gains from better clipping level management.
• At 6-bit, AdaBits achieves comparable performance to full-precision models, with model size reduced by 4.74× and BitOPs reduced by 14.25×, outperforming width-multiplier scaling (0.35×) which only reduces size by 2.06×.
• The S-CL mechanism enables optimal clipping levels per bit-width, as visualized in AB-MobileNet V1, where higher bit-widths consistently use larger clipping levels, aligning with individual model behavior.