[Paper Review] FPTQ: Fine-grained Post-Training Quantization for Large Language Models

Qingyuan Li, Yifan Zhang+7 more|arXiv (Cornell University)|Aug 30, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes FPTQ, a novel fine-grained post-training quantization method for large language models that achieves state-of-the-art W4A8 (4-bit weights, 8-bit activations) performance without fine-tuning. By combining layerwise activation quantization with a novel logarithmic equalization technique for difficult layers and fine-grained weight quantization, FPTQ enables efficient INT8 inference with INT4 weight storage, significantly improving memory and computation efficiency while preserving model accuracy across BLOOM, LLaMA, and LLaMA-2 benchmarks.

ABSTRACT

In the era of large-scale language models, the substantial parameter size poses significant challenges for deployment. Being a prevalent compression technique, quantization has emerged as the mainstream practice to tackle this issue, which is mainly centered on two recipes W8A8 and W4A16 (i.e. weights and activations in such bit widths). In this study, we propose a novel W4A8 post-training quantization method for the available open-sourced LLMs, which combines the advantages of both two recipes. Therefore, we can leverage the benefit in the I/O utilization of 4-bit weight quantization and the acceleration due to 8-bit matrix computation. Nevertheless, the W4A8 faces notorious performance degradation. As a remedy, we involve layerwise activation quantization strategies which feature a novel logarithmic equalization for most intractable layers, and we combine them with fine-grained weight quantization. Without whistles and bells, we eliminate the necessity for further fine-tuning and obtain the state-of-the-art W4A8 quantized performance on BLOOM, LLaMA, and LLaMA-2 on standard benchmarks. We confirm that the W4A8 quantization is achievable for the deployment of large language models, fostering their wide-spreading real-world applications.

Motivation & Objective

• To address the performance degradation commonly seen in W4A8 post-training quantization for large language models.
• To enable efficient deployment of LLMs on resource-constrained devices by optimizing both memory access and computation throughput.
• To eliminate the need for fine-tuning while maintaining state-of-the-art accuracy in W4A8 quantization.
• To develop a generalizable, inference-friendly quantization scheme that harmonizes 4-bit weight storage with 8-bit activation computation.

Proposed method

• Proposes a layerwise quantization strategy that adapts to varying activation distribution difficulties across different layers of the model.
• Introduces a novel logarithmic activation equalization (LAE) technique to stabilize extreme activation outliers in difficult layers, making them more quantization-friendly.
• Applies fine-grained weight quantization to 4-bit weights while maintaining 8-bit activations for efficient matrix multiplication on modern hardware.
• Uses offline activation statistics to compute scaling factors without requiring model fine-tuning or gradient updates.
• Employs a hyper-parameterized suppression function to control activation suppression levels, improving robustness across diverse layers.
• Designs a strategy that avoids the performance drop seen in hybrid quantization schemes by decoupling weight and activation quantization decisions per layer.

Experimental results

Research questions

• RQ1 Can W4A8 post-training quantization achieve state-of-the-art performance on LLMs without fine-tuning?
• RQ2 How can activation distributions with extreme ranges be stabilized to reduce quantization error in low-bit settings?
• RQ3 Can a layerwise quantization strategy with logarithmic equalization outperform uniform or global quantization in W4A8 settings?
• RQ4 Does combining 4-bit weights with 8-bit activations yield better inference efficiency than W8A8 or W4A16 in practice?
• RQ5 Can the proposed method maintain high accuracy across diverse LLMs like BLOOM, LLaMA, and LLaMA-2 without model-specific adaptation?

Key findings

• FPTQ achieves state-of-the-art W4A8 performance on BLOOM-7B1, LLaMA-7B, and LLaMA-2-7B across standard benchmarks like MMLU and Common Sense QA without any fine-tuning.
• On LLaMA-7B, FPTQ maintains an average MMLU score of 73.42, only 1.09 points below the FP16 baseline, despite using 4-bit weights.
• On LLaMA-2-7B, FPTQ achieves an average MMLU score of 73.77, outperforming the W4A8 baseline and remaining within 0.91 points of the FP16 model.
• The ablation study shows that applying GPTQ fine-tuning after LAE reduces performance, confirming that FPTQ’s success lies in its post-training design without additional tuning.
• The method enables efficient INT8 inference with INT4 weight storage, reducing memory bandwidth pressure in self-decoding stages while accelerating computation in context-decoding stages.
• The logarithmic equalization technique effectively stabilizes activation distributions in outlier-prone layers, as evidenced by the improved quantization stability in Figure 1.