[Paper Review] Benchmarking the Reliability of Post-training Quantization: a Particular Focus on Worst-case Performance

Zhihang Yuan, Jiawei Liu+7 more|arXiv (Cornell University)|Mar 23, 2023

Image Enhancement Techniques4 citations

TL;DR

This paper introduces a systematic framework to benchmark the reliability of post-training quantization (PTQ) methods, focusing on worst-case performance under distribution shifts. It reveals that while average accuracy remains stable, individual categories—especially under low-bit quantization or noisy calibration data—suffer significant accuracy drops, exposing critical reliability gaps in state-of-the-art PTQ methods despite strong average performance.

ABSTRACT

Post-training quantization (PTQ) is a popular method for compressing deep neural networks (DNNs) without modifying their original architecture or training procedures. Despite its effectiveness and convenience, the reliability of PTQ methods in the presence of some extrem cases such as distribution shift and data noise remains largely unexplored. This paper first investigates this problem on various commonly-used PTQ methods. We aim to answer several research questions related to the influence of calibration set distribution variations, calibration paradigm selection, and data augmentation or sampling strategies on PTQ reliability. A systematic evaluation process is conducted across a wide range of tasks and commonly-used PTQ paradigms. The results show that most existing PTQ methods are not reliable enough in term of the worst-case group performance, highlighting the need for more robust methods. Our findings provide insights for developing PTQ methods that can effectively handle distribution shift scenarios and enable the deployment of quantized DNNs in real-world applications.

Motivation & Objective

• To investigate the reliability of PTQ methods under extreme test conditions such as distribution shift and data noise.
• To identify factors—calibration set distribution, quantization settings, and optimization algorithms—that impact worst-case performance.
• To develop a standardized framework for evaluating PTQ reliability beyond average accuracy, focusing on subgroup and category-level performance.
• To highlight the limitations of current PTQ methods in risk-sensitive applications where worst-case robustness is essential.

Proposed method

• Proposes a systematic evaluation framework that assesses PTQ performance across average metrics and worst-case subgroups (e.g., rare or out-of-distribution categories).
• Employs diverse datasets (CIFAR-10, ImageNet) and network architectures (ResNet20, MobileNetV2) to evaluate reliability under varying conditions.
• Tests multiple calibration set compositions, including distribution shifts and injected noise, to analyze their impact on quantized model performance.
• Evaluates four optimization algorithms: grid search, Adaround, BRECQ, and QDrop, comparing their average and per-class accuracy stability.
• Uses statistical analysis across 50 random seeds to measure standard deviation of per-class accuracy, quantifying reliability variance.
• Introduces a benchmarking pipeline that enables reproducible, standardized evaluation of PTQ reliability across models, tasks, and datasets.

Experimental results

Research questions

• RQ1 How does variation in the calibration set's distribution affect the worst-case performance of quantized models?
• RQ2 How do different quantization settings (e.g., bit-width) influence performance degradation on specific categories?
• RQ3 How do different PTQ optimization algorithms compare in terms of reliability across individual classes, despite similar average accuracy?
• RQ4 To what extent does data augmentation or sampling strategy in calibration affect worst-case robustness?
• RQ5 Can gradient-based PTQ methods maintain reliability under distribution shift, even when they achieve high average accuracy?

Key findings

• Significant accuracy drops occur on individual categories after quantization, with some classes dropping by over 10 percentage points, indicating poor worst-case reliability.
• While average accuracy remains stable under calibration set distribution shifts or noise, per-class accuracy shows high sensitivity, especially in low-bit regimes.
• Reducing bit-width from 6 to 4 bits leads to substantial performance degradation on specific classes, such as a 15.5% drop on Class 4 in CIFAR-10 with W4A4 quantization.
• Gradient-based methods like QDrop achieve high average accuracy (72.1% on ImageNet W6A6), but exhibit high per-class standard deviation (up to ±2.77), indicating low reliability.
• On ImageNet, QDrop achieves 68.8% accuracy on Class 1, compared to 70.6% with grid search, showing that superior average performance does not guarantee robustness.
• The standard deviation of per-class accuracy across 50 runs is highest for QDrop (up to ±2.95), confirming that gradient-based methods are less reliable in worst-case scenarios despite strong average results.