[Paper Review] Efficient Post-training Quantization with FP8 Formats

Haihao Shen, Naveen Mellempudi+4 more|arXiv (Cornell University)|Sep 26, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a unified, scalable post-training quantization framework using FP8 formats (E5M2, E4M3, E3M4) to improve model accuracy and workload coverage over INT8, especially for large language models and vision tasks. It demonstrates that E4M3 excels in NLP (96.32% coverage) and E3M4 in vision (78.95% coverage), with mixed FP8 formats and dynamic quantization further boosting accuracy.

ABSTRACT

Recent advances in deep learning methods such as LLMs and Diffusion models have created a need for improved quantization methods that can meet the computational demands of these modern architectures while maintaining accuracy. Towards this goal, we study the advantages of FP8 data formats for post-training quantization across 75 unique network architectures covering a wide range of tasks, including machine translation, language modeling, text generation, image classification, generation, and segmentation. We examine three different FP8 representations (E5M2, E4M3, and E3M4) to study the effects of varying degrees of trade-off between dynamic range and precision on model accuracy. Based on our extensive study, we developed a quantization workflow that generalizes across different network architectures. Our empirical results show that FP8 formats outperform INT8 in multiple aspects, including workload coverage (92.64% vs. 65.87%), model accuracy and suitability for a broader range of operations. Furthermore, our findings suggest that E4M3 is better suited for NLP models, whereas E3M4 performs marginally better than E4M3 on computer vision tasks. The code is publicly available on Intel Neural Compressor: https://github.com/intel/neural-compressor.

Motivation & Objective

• To address the limitations of INT8 quantization in handling outliers and dynamic range, especially in modern models like LLMs and diffusion networks.
• To evaluate the effectiveness of three FP8 formats (E5M2, E4M3, E3M4) in post-training quantization across diverse model architectures.
• To develop a generalized, scalable quantization workflow applicable across NLP, computer vision, and multimodal tasks.
• To identify optimal FP8 formats and quantization strategies (e.g., mixed formats, dynamic quantization) that maximize accuracy and coverage.

Proposed method

• Proposes a unified post-training quantization workflow for FP8 formats, supporting both symmetric and asymmetric quantization with per-channel scaling.
• Employs three FP8 formats—E5M2 (wide dynamic range), E4M3 (balanced precision/dynamic range), and E3M4 (high precision, limited range)—to study trade-offs.
• Applies mixed FP8 formats by assigning E4M3 to activations and E3M4 to weights in NLP models to reduce quantization error.
• Uses data augmentation during calibration to improve BatchNorm statistics and model accuracy, especially with small calibration sets.
• Evaluates both static and dynamic quantization schemes, with dynamic quantization improving accuracy on NLP models.
• Introduces automatic model tuning based on accuracy-driven selection of quantization hyperparameters and formats.

Figure 1: ( left ) Histogram of the tensor $X\sim\mathcal{N}(\mu=0.0,\,\sigma^{2}=0.5)$ , that contains a small number ( 1%) of outliers uniformly distributed between -6.0 to 6.0. ( center ) Distribution of quantized values represented by E5M2, E4M3, E3M4 and INT8 data formats. ( right ) Overall qua

Experimental results

Research questions

• RQ1 How do different FP8 formats (E5M2, E4M3, E3M4) compare to INT8 in terms of model accuracy and workload coverage across diverse deep learning models?
• RQ2 What is the optimal FP8 format for NLP models versus computer vision models in post-training quantization?
• RQ3 Can mixed FP8 formats (e.g., E4M3 for activations, E3M4 for weights) improve quantization accuracy beyond single-format approaches?
• RQ4 How does dynamic quantization compare to static quantization when using FP8 formats in post-training quantization?
• RQ5 What calibration strategies (e.g., data augmentation) best preserve model accuracy for BatchNorm layers under FP8 quantization?

Key findings

• FP8 quantization achieves 92.64% workload coverage across 75 models, significantly outperforming INT8’s 65.87% coverage.
• E4M3 achieves 96.32% coverage on NLP models, outperforming E3M4 (92.11%), indicating its superiority for NLP workloads.
• E3M4 achieves 78.95% coverage on computer vision models, slightly outperforming E4M3 (73.68%) in this domain.
• Mixed FP8 formats (E4M3 for activations, E3M4 for weights) improve accuracy on BERT-base (MRPC: 0.9069 vs. 0.9050) and Funnel (MRPC: 0.9233 vs. 0.9207).
• Dynamic quantization improves accuracy by up to +0.98% on BERT-Large (RTE) with E4M3, demonstrating its benefit for NLP models.
• Using 3K training-augmented calibration samples with data augmentation preserves accuracy better than inference-only or smaller sets, especially for BatchNorm layers.

Figure 2: Standard Quantization Scheme : default configuration for broad set of operations across different workloads, Extended Quantization Scheme : configuration for additional operator coverage (Ex: LayerNorm, BatchNorm & element-wise), mixed FP8 formats, dynamic quantization, BatchNorm Calibrati