[Paper Review] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Elias Frantar, Saleh Ashkboos+2 more|arXiv (Cornell University)|Oct 31, 2022

Topic Modeling128 citations

TL;DR

GPTQ is a one-shot post-training quantization method that accurately compresses large GPT/OPT models to 3–4 bits per weight, enabling single-GPU inference and meaningful speedups without substantial accuracy loss.

ABSTRACT

Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.

Motivation & Objective

• Motivate reducing inference memory and compute for extremely large GPT/OPT models.
• Develop a one-shot, data-efficient quantization method with high accuracy at very low bit-widths.
• Scale quantization to models with hundreds of billions of parameters within practical compute budgets.
• Demonstrate end-to-end inference speedups and practical deployment on commodity GPUs.

Proposed method

• Builds on Optimal Brain Quantization (OBQ) and introduces major scalability improvements for large models.
• Quantizes weights layer-by-layer using an approximate second-order objective based on the Hessian inverse information.
• Reframes OBQ to quantize weights in blocks and reuse a fixed inverse Hessian structure to reduce runtime from cubic to near-quadratic in layer dimensions.
• Employs lazy batch-updates to improve GPU utilization and memory throughput.
• Uses a Cholesky reformulation and dampening to maintain numerical stability at scale.
• Provides a full GPTQ algorithm with practical optimizations and pseudocode for quantizing weight matrices.

Experimental results

Research questions

• RQ1 Can post-training quantization achieve high accuracy for models with hundreds of billions of parameters at 3–4 bit weights?
• RQ2 What algorithmic changes enable scalable, fast, and accurate one-shot quantization for giant language models?
• RQ3 How does GPTQ compare to existing post-training quantization methods in terms of perplexity, runtime, and practical deployment on GPUs?
• RQ4 What are the limits of extreme quantization (2-bit or ternary) for large LLMs and how can grouping help?
• RQ5 What are the practical implications for end-to-end inference speedups and hardware utilization when deploying GPTQ-quantized models?

Key findings

• GPTQ quantizes OPT-175B and BLOOM-176B to 3–4 bits with negligible perplexity loss compared to uncompressed baselines.
• Quantization runtimes reach approximately four GPU hours for 175B-scale models on a single GPU, enabling single-GPU generative inference.
• GPTQ outperforms RTN-style rounding at 3–4 bits, maintaining workable perplexity on 175B models where RTN collapses at 3-bit.
• Group-wise quantization (e.g., g1024, g128) further improves accuracy, especially at 3-bit and even enables 2-bit/ternary regimes with reasonable loss.
• End-to-end speedups are demonstrated: up to ~3.25x speedup on A100 and ~4.5x on A6000 for 3-bit OPT-175B generation kernels, plus memory-fit single-GPU inference for 3-bit models.
• The method remains compatible with various quantization grids and grouping schemes, enabling flexible tradeoffs between accuracy and compression.