[Paper Review] LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Tim Dettmers, Michael Lewis+2 more|arXiv (Cornell University)|Aug 15, 2022

Ferroelectric and Negative Capacitance Devices112 citations

TL;DR

Introduces LLM.int8(), a two-part quantization approach combining vector-wise 8-bit quantization with mixed-precision decomposition to enable 8-bit inference for transformers up to 175B parameters without accuracy degradation; opens access to large models on consumer GPUs.

ABSTRACT

Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.

Motivation & Objective

• Motivate memory reduction in large-scale transformer inference and quantify impact on performance.
• Develop an 8-bit quantization method that preserves full precision across multi-billion parameter models.
• Identify and quantify emergent large-magnitude outlier features that challenge quantization at scale.
• Propose a mixed-precision decomposition to isolate outliers without sacrificing 8-bit efficiency.
• Provide open-source tools and integration to enable widespread adoption.

Proposed method

• Propose vector-wise 8-bit quantization with per-inner-product normalization constants to improve precision.
• Introduce mixed-precision decomposition that isolates rare outlier feature dimensions into a 16-bit path while remaining 99.9% in 8-bit.
• Decompose matrix multiplication into outlier (16-bit) and regular (8-bit) parts and denormalize with outer-product constants.
• Analyze emergent large-magnitude outlier features and their impact on quantization as model size grows.
• Evaluate on perplexity (C4) across models 125M–13B and zeroshot accuracy on OPT models up to 175B.
• Open-source bitsandbytes with Hugging Face Transformers integration.

Experimental results

Research questions

• RQ1 How large can 8-bit quantization preserve model performance for transformer inference?
• RQ2 What are the emergent outlier features at scale, and how do they affect quantization accuracy?
• RQ3 Can mixed-precision decomposition maintain full 16-bit accuracy for outliers while using 8-bit quantization for the rest?
• RQ4 Does vector-wise quantization outperform other 8-bit quantization schemes as model size increases?
• RQ5 What are the practical implications for deploying 175B+ models on consumer GPUs?

Key findings

• LLM.int8() preserves perplexity across models from 125M to 13B, unlike absmax/row-wise/zeropoint baselines which degrade with size.
• For zeroshot tasks on OPT models up to 175B, LLM.int8() maintains full 16-bit performance, while baseline 8-bit methods degrade.
• Emergent large-magnitude outlier features appear around 6.7B parameters; 150k outliers per sequence are concentrated in about 6 feature dimensions.
• Mixed-precision decomposition (16-bit for outliers, 8-bit for the rest) enables zero-degradation inference up to 175B parameters.
• Memory reduction is substantial (about 2x for BLOOM-176B) with modest runtime impact, and 8-bit inference becomes feasible on consumer GPUs for very large models.