[论文解读] FP8 versus INT8 for efficient deep learning inference

Mart van Baalen, Andrey N. Kuzmin+9 more|arXiv (Cornell University)|Mar 31, 2023

Ferroelectric and Negative Capacitance Devices被引用 24

一句话总结

论文分析 FP8 与 INT8 在设备上高效推理中的比较，显示 FP8 在硬件效率和准确性方面通常不如 INT8，除非在某些含异常值较多的 transformer 案例；总体而言 INT8 仍然是推理效率的首选。

ABSTRACT

Recently, the idea of using FP8 as a number format for neural network training has been floating around the deep learning world. Given that most training is currently conducted with entire networks in FP32, or sometimes FP16 with mixed-precision, the step to having some parts of a network run in FP8 with 8-bit weights is an appealing potential speed-up for the generally costly and time-intensive training procedures in deep learning. A natural question arises regarding what this development means for efficient inference on edge devices. In the efficient inference device world, workloads are frequently executed in INT8. Sometimes going even as low as INT4 when efficiency calls for it. In this whitepaper, we compare the performance for both the FP8 and INT formats for efficient on-device inference. We theoretically show the difference between the INT and FP formats for neural networks and present a plethora of post-training quantization and quantization-aware-training results to show how this theory translates to practice. We also provide a hardware analysis showing that the FP formats are somewhere between 50-180% less efficient in terms of compute in dedicated hardware than the INT format. Based on our research and a read of the research field, we conclude that although the proposed FP8 format could be good for training, the results for inference do not warrant a dedicated implementation of FP8 in favor of INT8 for efficient inference. We show that our results are mostly consistent with previous findings but that important comparisons between the formats have thus far been lacking. Finally, we discuss what happens when FP8-trained networks are converted to INT8 and conclude with a brief discussion on the most efficient way for on-device deployment and an extensive suite of INT8 results for many models.

研究动机与目标

• 推动评估 FP8 作为设备端推理相对于 INT8 的潜在替代方案。
• 量化 FP8 与 INT8 在标准深度学习加速器中的硬件面积和能耗影响。
• 在多样化模型上评估 PTQ 和 QAT 的性能。
• 就 FP8 与 INT8 权衡给出设备端部署策略的指导。

提出的方法

• 将 FP8 格式定义为具有 4 或 5 个指数位（FP8-E4、FP8-E5），并在硬件与精度方面与 INT8 进行比较。
• 理论分析累加器实现（定点 Kulisch 与浮点）并估计门数/面积影响。
• 在广泛的模型集合（计算机视觉、自然语言处理、分割、3D）上进行 PTQ 和 QAT 实验以比较精度。
• 考察权重/激活分布中的离群值如何影响量化性能。
• 将 FP8 训练的网络转换为 INT8 以评估转换后的精度影响。

Figure 1 : A schematic overview of a deep learning accelerator. Weights and activations are stored in memory and moved to the MatMul calculation unit. The bit-width matters for both latency and energy consumed for transferring the data. The calculation unit does a matrix multiplication; here, both t

实验结果

研究问题

• RQ1 在设备端推理中，FP8（含不同指数位）的硬件效率（面积/功耗）与 INT8 相比如何？
• RQ2 在 PTQ 和 QAT 下，哪些格式（INT8 与 FP8-E4/FP8-E3/FP8-E2）在常见 DL 任务中提供最佳推理精度？
• RQ3 在实际应用中，FP8 网络是否需要特殊处理（如 transformer 特定的调整）以实现有竞争力的性能？
• RQ4 将 FP8 训练的网络转为 INT8 部署有什么影响？
• RQ5 在实际边缘部署中，FP8 更适合用于训练（梯度）而非推理吗？

主要发现

• FP8-E4 相对于 INT8 的硬件成本显著更高，在使用 FP32 或 FP16 累加时，对于相似的累加器，FP8-E4 需要比 INT8 多超过 50% 的门数。
• 在激活带宽主导的网络中，FP16/FP8 激活可能成为瓶颈，降低 FP8 相对 INT8 的潜在加速。
• PTQ 结果表明在良好行为的（高斯型）网络上，INT8 通常优于 FP8-E4/E5，而 FP8-E4/E5 在具有大量离群值的网络上（如某些 transformer 层）可能超过。
• QAT 结果通常在各格式下恢复到 FP32 的性能，对于许多 CV 网络 INT8 常提供最佳结果，FP8-E3 经常超过 FP8-E4。
• 在大多数网络中，FP8-E5 不适合推理，FP8-E4 在 QAT 后很少是最佳选择；INT8 或 FP8-E3/E2 提供更稳健的精度。
• W4A8（4 位权重，8 位激活）在某些情况下可匹配甚至超过 FP8-E4，表明低精度格式可能带来潜在的效率提升。
• 将 FP8-E4 训练的模型转换为 INT8 常常保持或略微提高某些体系结构的准确性，表明 FP8 训练可能并非实现具有竞争力的 INT8 性能所必需。

(a) Fixed-point accumulation