[Paper Review] GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

Ali Hadi Zadeh, Isak Edo+2 more|arXiv (Cornell University)|May 7, 2020

Topic Modeling50 references157 citations

TL;DR

GOBO is a post-training quantization technique that compresses 99.9% of 32-bit floating-point parameters in attention-based NLP models like BERT to just 3 bits using a novel centroid selection algorithm, maintaining accuracy without fine-tuning. It enables 10× model size reduction and up to 7× speedup with 3× lower energy consumption in hardware accelerators by keeping weights quantized during computation and reducing off-chip memory traffic.

ABSTRACT

Attention-based models have demonstrated remarkable success in various natural language understanding tasks. However, efficient execution remains a challenge for these models which are memory-bound due to their massive number of parameters. We present GOBO, a model quantization technique that compresses the vast majority (typically 99.9%) of the 32-bit floating-point parameters of state-of-the-art BERT models and their variants to 3 bits while maintaining their accuracy. Unlike other quantization methods, GOBO does not require fine-tuning nor retraining to compensate for the quantization error. We present two practical hardware applications of GOBO. In the first GOBO reduces memory storage and traffic and as a result inference latency and energy consumption. This GOBO memory compression mechanism is plug-in compatible with many architectures; we demonstrate it with the TPU, Eyeriss, and an architecture using Tensor Cores-like units. Second, we present a co-designed hardware architecture that also reduces computation. Uniquely, the GOBO architecture maintains most of the weights in 3b even during computation, a property that: (1) makes the processing elements area efficient, allowing us to pack more compute power per unit area, (2) replaces most multiply-accumulations with additions, and (3) reduces the off-chip traffic by amplifying on-chip memory capacity.

Motivation & Objective

• To address the high memory and energy costs of attention-based NLP models like BERT, which are memory-bound due to large parameter counts.
• To develop a quantization method that maintains model accuracy without requiring fine-tuning or retraining after quantization.
• To enable low-latency, energy-efficient inference by compressing model parameters and co-designing a hardware accelerator that operates directly on quantized weights.
• To demonstrate hardware compatibility and performance gains across multiple architectures, including TPU, Eyeriss, and Tensor Core-like units.
• To show that keeping weights in 3-bit format during computation improves area efficiency, reduces off-chip traffic, and replaces most multiply-accumulates with additions.

Proposed method

• GOBO identifies that 99.9% of weights in each layer follow a Gaussian distribution, while 0.1% are outliers.
• It stores outlier weights in full 32-bit precision and uses a dictionary of only 8 representative values (centroids) for the remaining weights.
• A novel, fast-converging centroid selection algorithm is used to group non-outlier weights into quantized bins, minimizing quantization error.
• The method avoids expanding quantized weights back to 32-bit floating-point values during computation, enabling on-chip processing in 3-bit format.
• It co-designs a hardware accelerator that performs computations directly on 3-bit weights, replacing most multiply-accumulates with additions.
• The hardware design increases on-chip memory capacity and reduces off-chip data movement by maintaining compressed weights throughout.

Experimental results

Research questions

• RQ1 Can a post-training quantization method compress 99.9% of BERT’s 32-bit parameters to 3 bits without fine-tuning while preserving accuracy?
• RQ2 How does keeping weights in 3-bit format during computation improve energy efficiency and performance compared to traditional accelerators?
• RQ3 What is the impact of GOBO compression on inference latency and energy consumption when applied to off-chip memory?
• RQ4 How does the novel centroid selection algorithm compare to K-Means in convergence speed and accuracy?
• RQ5 Can a co-designed hardware accelerator using GOBO achieve higher performance and energy efficiency than existing Tensor Core-like architectures under iso-compute-area constraints?

Key findings

• GOBO maintains model accuracy on the challenging MNLI task from GLUE while quantizing 99.9% of weights to 3 bits, with no fine-tuning required.
• The centroid selection algorithm converges 9× faster than K-Means and reduces the number of required centroids by half, improving compression efficiency.
• GOBO off-chip memory compression reduces model footprint by 10×, translating to a 10× performance improvement on TPU.
• Under iso-compute-area constraints, the GOBO accelerator is 7× faster and consumes 3× less energy than a Tensor Core-like accelerator.
• With FP16 precision, GOBO is 2.85× faster and 6.8× more energy efficient than IBM’s Deep Learning Processor (DLP), and 2.4× more efficient than DLP+ with GOBO compression.
• The GOBO accelerator achieves 83% PE utilization, compared to 34% in IBM DLP, due to reduced off-chip memory traffic and on-chip memory amplification.