[Paper Review] Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming

Itay Hubara, Yury Nahshan+3 more|arXiv (Cornell University)|Jun 14, 2020

Domain Adaptation and Few-Shot LearningComputer Science25 references76 citations

TL;DR

The paper introduces AdaQuant for layer-wise optimization of weights and activations during post-training quantization, plus an integer-programming approach for per-layer bit-width allocation and BN tuning, achieving state-of-the-art results on vision and language models with very small calibration sets.

ABSTRACT

Lately, post-training quantization methods have gained considerable attention, as they are simple to use, and require only a small unlabeled calibration set. This small dataset cannot be used to fine-tune the model without significant over-fitting. Instead, these methods only use the calibration set to set the activations' dynamic ranges. However, such methods always resulted in significant accuracy degradation, when used below 8-bits (except on small datasets). Here we aim to break the 8-bit barrier. To this end, we minimize the quantization errors of each layer separately by optimizing its parameters over the calibration set. We empirically demonstrate that this approach is: (1) much less susceptible to over-fitting than the standard fine-tuning approaches, and can be used even on a very small calibration set; and (2) more powerful than previous methods, which only set the activations' dynamic ranges. Furthermore, we demonstrate how to optimally allocate the bit-widths for each layer, while constraining accuracy degradation or model compression by proposing a novel integer programming formulation. Finally, we suggest model global statistics tuning, to correct biases introduced during quantization. Together, these methods yield state-of-the-art results for both vision and text models. For instance, on ResNet50, we obtain less than 1\% accuracy degradation --- with 4-bit weights and activations in all layers, but the smallest two. We open-sourced our code.

Motivation & Objective

• Address the accuracy loss in post-training quantization at sub-8-bit precision.
• Develop a layer-wise optimization method (AdaQuant) that uses a small calibration set without overfitting.
• Propose an integer-programming formulation to optimally allocate bit-widths across layers under accuracy/pruning constraints.
• Introduce batch normalization tuning to mitigate biases introduced during quantization.
• Offer lightweight and advanced deployment pipelines suitable for different hardware scenarios.

Proposed method

• AdaQuant: a layer-by-layer joint optimization of weights and quantization parameters to minimize the MSE between full-precision and quantized outputs, allowing V to adjust W (Equation 2) and sequential vs parallel variants (Eq. 3-4).
• Integer programming formulation for mixed-precision bit allocation where each layer selects a (k,n) bit configuration to maximize end-to-end performance gain under a total accuracy degradation constraint (Equations 5a–5d).
• Batch Normalization Tuning (BNT): reconstruct BN, re-estimate running statistics on calibration data, and re-fuse BN into weights to correct statistics (Equations 6–8).
• Light vs Advanced pipelines: a light pipeline with no backward pass and an advanced pipeline that adds AdaQuant, IP, BN tuning, and bias tuning (Table 1).
• Evaluation across vision (ResNet variants, MobileNet-V2) and language (BERT on SQuAD) showing AdaQuant’s superiority over QAT-KLD and AdaRound, especially with small calibration sets.

Experimental results

Research questions

• RQ1 How can post-training quantization achieve sub-8-bit accuracy without extensive retraining or large calibration datasets?
• RQ2 Can a layer-wise optimization (AdaQuant) improve quantization accuracy by jointly tuning weights and quantization parameters across layers?
• RQ3 How can integer programming be used to optimally allocate bit-widths per layer under accuracy/throughput constraints?
• RQ4 What is the impact of batch-norm tuning on quantized models, and can BN statistics be effectively re-estimated to recover accuracy?
• RQ5 Do the proposed AdaQuant and IP-based methods yield state-of-the-art results for both vision and text models under practical calibration constraints?

Key findings

• AdaQuant achieves state-of-the-art post-training quantization across multiple models and tasks, including vision (ResNet, MobileNet-V2) and language (BERT) with small calibration sets.
• AdaQuant reduces quantization error by jointly optimizing weights, quantization parameters, and biases, outperforming AdaRound and QAT-based baselines, especially at very low calibration data.
• Per-layer IP bit allocation can further improve end-to-end performance by selecting mixed-precision configurations (e.g., 4-bit vs 8-bit) under a total degradation constraint, and even enabling a 2-bit layer in some cases to boost compression.
• Batch-Norm Tuning (BNT) effectively mitigates mean/variance shifts caused by quantization, yielding noticeable accuracy gains with only a few forward passes on calibration data.
• Two deployment pipelines are proposed: a light pipeline (no backward pass) and an advanced pipeline (AdaQuant + IP + BN tuning + bias tuning), both achieving strong results without retraining.