[論文レビュー] Efficient 8-Bit Quantization of Transformer Neural Machine Language\n Translation Model

Aishwarya Bhandare, Vamsi Sripathi+5 more|arXiv (Cornell University)|Jun 2, 2019

Advanced Neural Network Applications被引用数 56

ひとこと要約

本論文は、KLダイバージェンスに基づく適合閾値とCPU最適化のMKL/VNNIを用いて、訓練済みのTransformer翻訳モデルをINT8へ量子化し、0.5 BLEUポイント未満の低下と最大で全体スループットを1.5倍程度向上させる。

ABSTRACT

In this work, we quantize a trained Transformer machine language translation\nmodel leveraging INT8/VNNI instructions in the latest Intel$^\\circledR$\nXeon$^\\circledR$ Cascade Lake processors to improve inference performance while\nmaintaining less than 0.5$\\%$ drop in accuracy. To the best of our knowledge,\nthis is the first attempt in the industry to quantize the Transformer model.\nThis has high impact as it clearly demonstrates the various complexities of\nquantizing the language translation model. We present novel quantization\ntechniques directly in TensorFlow to opportunistically replace 32-bit floating\npoint (FP32) computations with 8-bit integers (INT8) and transform the FP32\ncomputational graph. We also present a bin-packing parallel batching technique\nto maximize CPU utilization. Overall, our optimizations with INT8/VNNI deliver\n1.5X improvement over the best FP32 performance. Furthermore, it reveals the\nopportunities and challenges to boost performance of quantized deep learning\ninference and establishes best practices to run inference with high efficiency\non Intel CPUs.\n

研究の動機と目的

• 推論効率を改善するためにTransformer翻訳モデルの量子化を動機づけ、精度の大幅な低下を避ける。
• KLダイバージェンスに基づく較正と選択的量子化を用いて精度を維持する量子化ワークフローを開発する。
• Intel CPU上の推論時にCPUスルー率を最大化するためにカーネルとグラフ構造を最適化する。

提案手法

• FP32テンソルとINT8テンソル分布のヒストグラム間のKLダイバージェンスに基づく較正プロセスを用いてFP32テンソルをINT8へ量子化する。
• 符号付き/符号なしINT8表現の最小/最大閾値を決定するために、対称/独立/共役の較正モードを使用する。
• MKLでFP32のMatMulをQuantizedMatMulに置換し、非ゼロオフセットに対処してスピードアップを達成する。
• データを量子化し量子化を再配置してデータコピーを最小化することでGatherNdを最適化する。
• 入力文をトークン数でソートし、マルチストリーム推論による並列バッチ処理を適用してCPU利用率を向上させる。
• 冗長な演算を排除し、計算グラフを融合/再形成してオーバーヘッドを減らしスループットを改善する。

実験結果

リサーチクエスチョン

• RQ1 Can Transformer translation models be quantized to 8-bit integers with less than 0.5 BLEU point degradation?
• RQ2 What calibration strategy (symmetric/independent/conjugate) best preserves translation quality when quantizing?
• RQ3 How can MKL INT8/VNNI kernels be leveraged and graph optimizations be applied to maximize CPU inference throughput for quantized Transformers?
• RQ4 What system-level techniques (input sorting, parallel batching) most effectively improve throughput on Intel CPUs?
• RQ5 What practical best practices emerge for quantizing Transformer models on CPU architectures?

主な発見

Mode	BLEU Score	Drop in Accuracy
Naïve quantization	NA	NA
Symmetric	27.30	0.38
Independent	27.33	0.35
Conjugate	27.26	0.421

• Quantizing a trained FP32 Transformer to INT8 maintained BLEU score within 0.5 points of the FP32 model.
• KL-divergence based calibration with independent thresholds gave the least accuracy drop among tested modes, though symmetric thresholds were chosen for practicality.
• MKL INT8/VNNI kernels yielded substantial speedups for MatMul, especially after correcting non-zero offset handling (average 2.4x across MatMuls).
• Gath erNd and related data copy overheads were reduced by quantizing data, cutting copy size by 3.8x and decreasing GatherNd execution time by 5x.
• Input data sorting by token count and parallel batching increased CPU utilization, achieving up to 1.5x throughput gains over best FP32 configurations in end-to-end tests.
• Overall throughput improved by up to 4.5x in INT8/VNNI scaling relative to baseline FP32 with optimizations; system-level optimizations raised FP32 performance by up to 3x, yielding 1.51x relative INT8 performance on best config.