[Paper Review] FP8 versus INT8 for efficient deep learning inference

Mart van Baalen, Andrey N. Kuzmin+9 more|arXiv (Cornell University)|Mar 31, 2023

Ferroelectric and Negative Capacitance Devices24 citations

TL;DR

The paper analyzes FP8 vs INT8 for efficient on-device inference, showing FP8 often underperforms INT8 in hardware efficiency and accuracy except in certain outlier-heavy transformer cases; overall INT8 remains the preferable choice for inference efficiency.

ABSTRACT

Recently, the idea of using FP8 as a number format for neural network training has been floating around the deep learning world. Given that most training is currently conducted with entire networks in FP32, or sometimes FP16 with mixed-precision, the step to having some parts of a network run in FP8 with 8-bit weights is an appealing potential speed-up for the generally costly and time-intensive training procedures in deep learning. A natural question arises regarding what this development means for efficient inference on edge devices. In the efficient inference device world, workloads are frequently executed in INT8. Sometimes going even as low as INT4 when efficiency calls for it. In this whitepaper, we compare the performance for both the FP8 and INT formats for efficient on-device inference. We theoretically show the difference between the INT and FP formats for neural networks and present a plethora of post-training quantization and quantization-aware-training results to show how this theory translates to practice. We also provide a hardware analysis showing that the FP formats are somewhere between 50-180% less efficient in terms of compute in dedicated hardware than the INT format. Based on our research and a read of the research field, we conclude that although the proposed FP8 format could be good for training, the results for inference do not warrant a dedicated implementation of FP8 in favor of INT8 for efficient inference. We show that our results are mostly consistent with previous findings but that important comparisons between the formats have thus far been lacking. Finally, we discuss what happens when FP8-trained networks are converted to INT8 and conclude with a brief discussion on the most efficient way for on-device deployment and an extensive suite of INT8 results for many models.

Motivation & Objective

• Motivate the assessment of FP8 as a potential alternative to INT8 for on-device inference.
• Quantify hardware-area and energy implications of FP8 vs INT8 in standard DL accelerators.
• Evaluate post-training quantization (PTQ) and quantization-aware training (QAT) performance across diverse models.
• Provide guidance on on-device deployment strategies given FP8 vs INT8 trade-offs.

Proposed method

• Define FP8 formats with 4 or 5 exponent bits (FP8-E4, FP8-E5) and compare to INT8 across hardware and accuracy.
• Theoretically analyze accumulator implementations (fixed-point Kulisch vs floating-point) and estimate gate-count/area impacts.
• Conduct PTQ and QAT experiments across a broad set of models (CV, NLP, segmentation, 3D) to compare accuracy.
• Examine how outliers in weight/activation distributions influence quantization performance.
• Convert FP8-trained networks to INT8 to assess post-conversion accuracy implications.

Figure 1 : A schematic overview of a deep learning accelerator. Weights and activations are stored in memory and moved to the MatMul calculation unit. The bit-width matters for both latency and energy consumed for transferring the data. The calculation unit does a matrix multiplication; here, both t

Experimental results

Research questions

• RQ1 How does FP8 (with varying exponent bits) compare to INT8 in terms of hardware efficiency (area/power) for on-device inference?
• RQ2 Which formats (INT8 vs FP8-E4/FP8-E3/FP8-E2) yield the best inference accuracy across common DL tasks under PTQ and QAT?
• RQ3 In practice, do FP8 networks require special handling (e.g., transformer-specific adjustments) to achieve competitive performance?
• RQ4 What are the implications of converting FP8-trained networks to INT8 for deployment?
• RQ5 Is FP8 more suitable for training (gradients) than for inference in real-world edge deployments?

Key findings

• FP8-E4 incurs significantly higher hardware costs than INT8, with FP8-E4 needing more than 50% more gates than INT8 for similar accumulators when using FP32 or FP16 accumulation.
• For activation-bandwidth dominated networks, FP16/FP8 activations can bottleneck, reducing potential FP8 speedups relative to INT8.
• PTQ results show INT8 often outperforms FP8-E4/E5 on well-behaved (Gaussian-like) networks, while FP8-E4/E5 can outperform on networks with substantial outliers (e.g., some transformer layers).
• QAT results generally recover FP32 performance across formats, with INT8 often providing the best results for many CV networks and FP8-E3 frequently outperforming FP8-E4.
• Across most networks, FP8-E5 is not favored for inference, and FP8-E4 is rarely the best choice after QAT; INT8 or FP8-E3/E2 provide more robust accuracy.
• W4A8 (4-bit weights, 8-bit activations) can match or exceed FP8-E4 in some cases, suggesting potential efficiency gains with lower-precision formats.
• Converting FP8-E4 trained models to INT8 often preserves or slightly improves accuracy for some architectures, indicating that FP8 training may not be strictly necessary for competitive INT8 performance.

(a) Fixed-point accumulation