[Paper Review] LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

Gunho Park, Baeseong Park+8 more|arXiv (Cornell University)|Jun 20, 2022

Ferroelectric and Negative Capacitance Devices20 citations

TL;DR

LUT-GEMM introduces a LUT-based matrix multiplication kernel that supports weight-only quantization (uniform or non-uniform via extended BCQ with bias) and full-precision activations, enabling substantial latency reductions and the possibility to run OPT-175B on a single GPU with minimal accuracy loss.

ABSTRACT

Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity. However, the growing size of NLP models introduces a memory wall problem during the generation phase. To mitigate this issue, recent efforts have focused on quantizing model weights to sub-4-bit precision while preserving full precision for activations, resulting in practical speed-ups during inference on a single GPU. However, these improvements primarily stem from reduced memory movement, which necessitates a resource-intensive dequantization process rather than actual computational reduction. In this paper, we introduce LUT-GEMM, an efficient kernel for quantized matrix multiplication, which not only eliminates the resource-intensive dequantization process but also reduces computational costs compared to previous kernels for weight-only quantization. Furthermore, we proposed group-wise quantization to offer a flexible trade-off between compression ratio and accuracy. The impact of LUT-GEMM is facilitated by implementing high compression ratios through low-bit quantization and efficient LUT-based operations. We show experimentally that when applied to the OPT-175B model with 3-bit quantization, LUT-GEMM substantially accelerates token generation latency, achieving a remarkable 2.1$ imes$ improvement on a single GPU when compared to OPTQ, which relies on the costly dequantization process.

Motivation & Objective

• Motivate memory- and energy-efficient inference for large-scale generative LMs by collapsing weight precision while preserving activation precision.
• Propose a weight-only quantization kernel that eliminates dequantization overhead for both uniform and non-uniform quantization.
• Develop LUT-GEMM based on extended binary-coding quantization (BCQ) with bias to represent diverse quantization schemes.
• Show that increasing compression (low-bit weights) reduces GPU count and latency without substantial accuracy loss in large LMs.
• Demonstrate practical speedups and single-GPU inference feasibility for very large models (e.g., OPT-175B).

Proposed method

• Extend BCQ with a bias term to represent both non-uniform and uniform quantization within a unified BCQ format.
• Introduce group-wise BCQ with a configurable group size g to trade off compression ratio and quantization error.
• Develop LUT-GEMM that uses LUT-based computations to replace redundant partial sums and avoid activation dequantization, enabling efficient weight-only quantized GEMMs.
• Architect LUT-GEMM to map to GPU threads with shared LUTs in fast on-chip memory and optimize tiling/hete steps (parameters per TB, mu=8, l, t_h, t_w, q, g).
• Formulate the single-batch LUT-GEMM operation as y = sum_{i=1}^q (A_i ∘ (B_i · x)), where A is FP16 scaling, B is FP16 binary, x is FP16 input, and ∘ is element-wise multiplication.
• Provide an empirical LUT Optimization (mu, g, q) guidance and discuss memory-footprint-driven latency.

Experimental results

Research questions

• RQ1 Can a weight-only quantization kernel with BCQ-based representations accelerate large LM inference without dequantizing activations?
• RQ2 How does extending BCQ with a bias term and enabling group-wise BCQ affect compression, latency, and accuracy for very large LMs?
• RQ3 What performance gains (latency) can be achieved for OPT-175B using LUT-GEMM compared to existing dequantization-based approaches (e.g., OPTQ)?
• RQ4 What are practical GPU-implementation considerations (thread configuration, LUT storage, memory footprint) that maximize LUT-GEMM throughput?

Key findings

• LUT-GEMM enables weight-only quantization with a flexible BCQ-based representation that accommodates both uniform and non-uniform quantization formats.
• Extending BCQ with a bias term and introducing group-wise quantization expands the quantization space, enabling better compression–latency trade-offs for large LMs.
• For OPT-175B, LUT-GEMM achieves 2.1× faster token-generation latency than OPTQ when using 3-bit BCQ weights on a single GPU, reducing the need for multiple GPUs.
• LUT-GEMM demonstrates that near single-GPU inference of OPT-175B is possible with minimal accuracy/ performance losses, addressing memory bottlenecks without heavy dequantization overhead.
• Latency scaling is closely tied to memory footprint; increasing group size g beyond a threshold yields latency close to row-wise BCQ while delivering higher compression opportunities.
• Empirical results show LUT-GEMM latency advantages across kernels (e.g., LUT-GEMM with BCQ3/INT3 at 0.1956 ms vs OPTQ FP16/INT3 at 0.3047 ms in Table 2).