[论文解读] ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

Zhewei Yao, Xiaoxia Wu+3 more|arXiv (Cornell University)|Mar 15, 2023

Topic Modeling被引用 15

一句话总结

本论文在模型家族（OPT 和 BLOOM）及规模上进行全面的后量化（PTQ）研究，比较权重量化、激活量化、以及权重-激活量化在 RTN、GPTQ、ZeroQuant 变体中的表现，并引入 LoRC 以在最小的尺寸增加下恢复模型质量。

ABSTRACT

Post-training quantization (PTQ) has emerged as a promising technique for mitigating memory consumption and computational costs in large language models (LLMs). However, a systematic examination of various quantization schemes, model families, and quantization bit precision has been absent from the literature. In this paper, we conduct a comprehensive analysis of these factors by investigating the effects of PTQ on weight-only, activation-only, and weight-and-activation quantization using diverse methods such as round-to-nearest (RTN), GPTQ, ZeroQuant, and their variants. We apply these methods to two distinct model families with parameters ranging from 125M to 176B. Our contributions include: (1) a sensitivity analysis revealing that activation quantization is generally more susceptible to weight quantization, with smaller models often outperforming larger models in terms of activation quantization; (2) an evaluation and comparison of existing PTQ methods to optimize model size reduction while minimizing the impact on accuracy, revealing that none of the current methods can achieve the original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation; (3) based on these insights, we propose an optimized method called Low-Rank Compensation (LoRC), which employs low-rank matrices to enhance model quality recovery with a minimal increase in model size.

研究动机与目标

• 评估 PTQ 在不同模型规模和家族下，在权重量化、激活量化，以及权重与激活量化情形下的表现。
• 评估现有 PTQ 方法（RTN、GPTQ、ZeroQuant 变体）在减小模型大小的同时保持精度的能力。
• 识别跨模型和不同规模的激活量化与权重量化之间的敏感性模式。
• 提出通过低秩补偿技术改进 PTQ，以恢复到 FP16 质量的性能。
• 按模型规模分组提供实用的量化指南。

提出的方法

• 使用 RTN、GPTQ、ZeroQuant 及其变体，对 OPT 和 BLOOM 模型（125M 到 176B）执行权重量化、激活量化以及权重与激活量化。
• 对激活量化与权重量化进行敏感性分析，包括对称量化/非对称量化以及按行/按-token 方案。
• 在优化配置下比较 PTQ 方法，以在最大化尺寸缩减的同时最小化困惑度衰减。
• 通过对量化误差 E = W - W_hat 使用 SVD 分解为低秩矩阵 U 和 V，将 LoRC（Low Rank Compens ation）引入以增强量化权重。
• 用 FGQ（细粒度量化）演示 LoRC 并量化参数开销；分析最佳低秩维度 m。
• 按模型规模和量化设置提供实用的量化建议。

实验结果

研究问题

• RQ1 Do LLMs of different sizes and pretraining data exhibit similar behavior under quantization?
• RQ2 Are existing PTQ methods effectively minimizing LLM sizes without sacrificing accuracy?
• RQ3 How do weight-only, activation-only, and weight-and-activation quantization compare across model families (OPT and BLOOM)?
• RQ4 Can LoRC improve model quality recovery with minimal size increase when combined with FGQ and PTQ?
• RQ5 What practical quantization settings are recommended for different model sizes?

主要发现

• Activation quantization is generally more sensitive to weight quantization across models; smaller models often outperform larger models in activation quantization.
• Existing PTQ methods struggle to reach original model quality for INT4 weight or INT4 weight with INT8 activation (W4A8) quantization.
• LoRC improves model quality with minimal parameter overhead by approximating quantization error with low-rank matrices; gains are larger when combined with FGQ.
• GPTQ tends to perform best for weight-only quantization, while ZeroQuant variants generally outperform for weight-and-activation quantization.
• Fine-grained quantization (FGQ) substantially reduces error, enabling Class -1 performance for larger models (≥10B) with 4-bit weight; activation block size and model size influence gains.
• LoRC can nearly recover FP16 quality for INT4 quantization, with optimal gains at low ranks (m ≈ 4–8).