[Paper Review] RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers

Zhikai Li, Junrui Xiao+2 more|arXiv (Cornell University)|Dec 16, 2022

CCD and CMOS Imaging Sensors4 citations

TL;DR

RepQ-ViT proposes a novel post-training quantization framework for Vision Transformers that decouples quantization accuracy from hardware efficiency via scale reparameterization. By initially applying complex quantizers—channel-wise for post-LayerNorm and log√2 for post-Softmax activations—and reparameterizing their scales to hardware-friendly layer-wise and log2 quantizers, RepQ-ViT achieves state-of-the-art 4-bit accuracy (e.g., 69.03% top-1 on DeiT-S) without reconstruction or hyperparameter tuning.

ABSTRACT

Post-training quantization (PTQ), which only requires a tiny dataset for calibration without end-to-end retraining, is a light and practical model compression technique. Recently, several PTQ schemes for vision transformers (ViTs) have been presented; unfortunately, they typically suffer from non-trivial accuracy degradation, especially in low-bit cases. In this paper, we propose RepQ-ViT, a novel PTQ framework for ViTs based on quantization scale reparameterization, to address the above issues. RepQ-ViT decouples the quantization and inference processes, where the former employs complex quantizers and the latter employs scale-reparameterized simplified quantizers. This ensures both accurate quantization and efficient inference, which distinguishes it from existing approaches that sacrifice quantization performance to meet the target hardware. More specifically, we focus on two components with extreme distributions: post-LayerNorm activations with severe inter-channel variation and post-Softmax activations with power-law features, and initially apply channel-wise quantization and log$\sqrt{2}$ quantization, respectively. Then, we reparameterize the scales to hardware-friendly layer-wise quantization and log2 quantization for inference, with only slight accuracy or computational costs. Extensive experiments are conducted on multiple vision tasks with different model variants, proving that RepQ-ViT, without hyperparameters and expensive reconstruction procedures, can outperform existing strong baselines and encouragingly improve the accuracy of 4-bit PTQ of ViTs to a usable level. Code is available at https://github.com/zkkli/RepQ-ViT.

Motivation & Objective

• To address the significant accuracy degradation in low-bit post-training quantization of Vision Transformers, especially in 4-bit settings.
• To overcome the trade-off between quantization accuracy and hardware efficiency in existing PTQ methods for ViTs.
• To decouple the quantization process (for accuracy) from inference (for efficiency) using scale reparameterization.
• To enable hardware-friendly inference without sacrificing model performance through explicit scale transformation.
• To achieve state-of-the-art results in 4-bit PTQ of ViTs with minimal calibration data and no retraining.

Proposed method

• RepQ-ViT decouples quantization and inference by using complex quantizers during calibration and simplified, hardware-friendly quantizers during inference.
• For post-LayerNorm activations with high inter-channel variation, channel-wise quantization is applied initially, then reparameterized to layer-wise quantization via adjustment of LayerNorm affine factors and subsequent layer weights.
• For post-Softmax activations with power-law distributions, log√2 quantization is used initially to better capture extreme values, then reparameterized to log2 quantization for efficient bit-shifting inference.
• Scale reparameterization is achieved through equivalent transformations that preserve accuracy while enabling efficient hardware computation.
• The method requires only 32 calibration samples and avoids expensive reconstruction or hyperparameter tuning.
• The framework is theoretically grounded and maintains high robustness with only minor accuracy or computational overhead.

Experimental results

Research questions

• RQ1 Can quantization accuracy and hardware efficiency be decoupled in post-training quantization of Vision Transformers?
• RQ2 Can complex quantizers be effectively reparameterized into hardware-friendly forms without significant accuracy loss?
• RQ3 Does scale reparameterization improve 4-bit PTQ performance on ViTs compared to existing methods?
• RQ4 Can log√2 quantization better represent power-law distributed attention scores than log2 quantization?
• RQ5 Can the proposed method achieve state-of-the-art results with minimal calibration data and no retraining?

Key findings

• RepQ-ViT achieves 69.03% top-1 accuracy on DeiT-S with 4-bit weights and activations, significantly outperforming prior methods such as FQ-ViT (0.10%) and PTQ4ViT (34.08%).
• On Swin-S, RepQ-ViT attains 79.45% top-1 accuracy in 4-bit PTQ, surpassing the strong baseline PTQ4ViT (76.09%).
• The method requires only 32 calibration samples, reducing data needs compared to PTQ4ViT (32 samples) and FQ-ViT (1000 samples), while achieving higher accuracy.
• The inference efficiency is maintained via reparameterization to log2 quantization, enabling bit-shifting operations with only a slight computational overhead.
• The scale reparameterization process incurs only a 1.25% accuracy drop compared to channel-wise quantization, while enabling hardware-friendly layer-wise quantization.
• The method is robust and generalizable, showing consistent improvements across multiple ViT variants and vision tasks without hyperparameter tuning or reconstruction.