[Paper Review] PD-Quant: Post-Training Quantization based on Prediction Difference Metric

Jiawei Liu, Lin Niu+4 more|arXiv (Cornell University)|Dec 14, 2022

Advanced Neural Network Applications4 citations

TL;DR

PD-Quant proposes a post-training quantization method that improves low-bit model accuracy by optimizing quantization parameters using prediction difference—specifically, the KL divergence between full-precision and quantized model outputs. It introduces Distribution Correction to mitigate overfitting on small calibration sets, achieving state-of-the-art results, such as 53.14% top-1 accuracy for ResNet-18 with 2-bit weights and activations.

ABSTRACT

Post-training quantization (PTQ) is a neural network compression technique that converts a full-precision model into a quantized model using lower-precision data types. Although it can help reduce the size and computational cost of deep neural networks, it can also introduce quantization noise and reduce prediction accuracy, especially in extremely low-bit settings. How to determine the appropriate quantization parameters (e.g., scaling factors and rounding of weights) is the main problem facing now. Existing methods attempt to determine these parameters by minimize the distance between features before and after quantization, but such an approach only considers local information and may not result in the most optimal quantization parameters. We analyze this issue and ropose PD-Quant, a method that addresses this limitation by considering global information. It determines the quantization parameters by using the information of differences between network prediction before and after quantization. In addition, PD-Quant can alleviate the overfitting problem in PTQ caused by the small number of calibration sets by adjusting the distribution of activations. Experiments show that PD-Quant leads to better quantization parameters and improves the prediction accuracy of quantized models, especially in low-bit settings. For example, PD-Quant pushes the accuracy of ResNet-18 up to 53.14% and RegNetX-600MF up to 40.67% in weight 2-bit activation 2-bit. The code is released at https://github.com/hustvl/PD-Quant.

Motivation & Objective

• To address the performance drop in post-training quantization (PTQ) at extremely low bit-widths, especially 2-bit, where quantization noise severely affects accuracy.
• To overcome the limitations of local metrics (e.g., MSE or cosine distance) in selecting optimal quantization scaling factors and rounding values.
• To reduce overfitting in PTQ caused by limited calibration data by correcting activation distributions to match batch normalization statistics.
• To develop a method that optimizes quantization parameters using global information from model prediction differences rather than local feature reconstruction.

Proposed method

• PD-Quant uses the KL divergence between the full-precision model’s and the quantized model’s final predictions as the primary metric to guide quantization parameter optimization.
• It jointly optimizes activation scaling factors and weight rounding values by minimizing the prediction difference loss, which captures global model behavior rather than local feature shifts.
• The method introduces Distribution Correction (DC), which adjusts the mean and variance of calibration set activations to match those stored in batch normalization layers, improving generalization.
• The optimization is performed layer-wise using a differentiable search strategy over scaling factors and rounding values, with a focus on low-bit settings.
• PD-Quant is applied post-training without retraining, relying only on a small set of unlabeled calibration samples.
• The approach is evaluated on ResNet-18, RegNetX-600MF, and MobileNetV2, showing consistent gains across models and bit-widths.

Experimental results

Research questions

• RQ1 Can prediction difference, measured via KL divergence between full-precision and quantized model outputs, lead to better quantization parameters than local metrics like MSE or cosine distance?
• RQ2 Does using global prediction difference information improve quantization accuracy, especially in low-bit settings such as 2-bit?
• RQ3 Can distribution correction of calibration set activations mitigate overfitting in PTQ when using limited calibration data?
• RQ4 How does the proposed method compare to existing PTQ baselines in terms of accuracy and computational cost at low bit-widths?

Key findings

• PD-Quant achieves 53.14% top-1 accuracy on ImageNet for ResNet-18 with 2-bit weights and 2-bit activations, setting a new state-of-the-art for PTQ in this setting.
• For RegNetX-600MF, PD-Quant reaches 40.67% top-1 accuracy under the same 2-bit weight and activation configuration, significantly outperforming prior PTQ methods.
• The use of prediction difference via KL divergence as the optimization objective leads to more accurate quantization parameters than local metrics such as MSE or cosine distance.
• Distribution Correction (DC) improves model generalization and reduces overfitting, with minimal hyperparameter tuning, as shown by stable performance across different λc values.
• PD-Quant’s additional inference time cost is acceptable—only 1.11 hours on an RTX A6000 for ResNet-18, compared to 120 hours for QAT methods like LSQ.
• The method is robust to low-bit weight quantization, though it shows limited improvement for weight scaling factors, indicating that activation quantization is more sensitive to prediction-based optimization.