[Paper Review] Efficient Execution of Quantized Deep Learning Models: A Compiler Approach

Animesh Jain, Shoubhik Bhattacharya+3 more|arXiv (Cornell University)|Jun 18, 2020

Advanced Neural Network Applications24 references17 citations

TL;DR

This paper proposes a compiler-level solution using a new Quantized Neural Network (QNN) dialect to efficiently execute pre-quantized 8-bit integer deep learning models across diverse hardware platforms. By extending deep learning compilers like Apache TVM with a quantization context, the approach enables high-performance inference on Intel CPUs, NVIDIA GPUs, and ARM edge devices, achieving speedups of up to 2.35× over FP32 execution while matching framework-specific optimizations across platforms.

ABSTRACT

A growing number of applications implement predictive functions using deep learning models, which require heavy use of compute and memory. One popular technique for increasing resource efficiency is 8-bit integer quantization, in which 32-bit floating point numbers (fp32) are represented using shorter 8-bit integer numbers. Although deep learning frameworks such as TensorFlow, TFLite, MXNet, and PyTorch enable developers to quantize models with only a small drop in accuracy, they are not well suited to execute quantized models on a variety of hardware platforms. For example, TFLite is optimized to run inference on ARM CPU edge devices but it does not have efficient support for Intel CPUs and Nvidia GPUs. In this paper, we address the challenges of executing quantized deep learning models on diverse hardware platforms by proposing an augmented compiler approach. A deep learning compiler such as Apache TVM can enable the efficient execution of model from various frameworks on various targets. Many deep learning compilers today, however, are designed primarily for fp32 computation and cannot optimize a pre-quantized INT8 model. To address this issue, we created a new dialect called Quantized Neural Network (QNN) that extends the compiler's internal representation with a quantization context. With this quantization context, the compiler can generate efficient code for pre-quantized models on various hardware platforms. As implemented in Apache TVM, we observe that the QNN-augmented deep learning compiler achieves speedups of 2.35x, 2.15x, 1.35x and 1.40x on Intel Xeon Cascade Lake CPUs, Nvidia Tesla T4 GPUs, ARM Raspberry Pi3 and Pi4 respectively against well optimized fp32 execution, and comparable performance to the state-of-the-art framework-specific solutions.

Motivation & Objective

• To address the inefficiency of existing deep learning frameworks in executing pre-quantized INT8 models across heterogeneous hardware platforms.
• To reduce developer effort by enabling a unified deployment pipeline for quantized models across multiple frameworks and hardware targets.
• To overcome the limitations of framework-specific kernel libraries and operator implementations that hinder cross-platform portability of quantized models.
• To enable deep learning compilers to natively optimize and generate efficient code for pre-quantized models using a standardized quantization-aware representation.

Proposed method

• Introduce a new graph-level dialect called Quantized Neural Network (QNN) that extends the compiler’s internal representation with a quantization context, including scale and zero-point parameters.
• Augment existing deep learning compilers, such as Apache TVM, with the QNN dialect to support compilation of pre-quantized models from frameworks like TensorFlow Lite, PyTorch, and MXNet.
• Leverage hardware-specific kernel libraries (e.g., Intel DNNL, CuDNN, ACL) through the QNN representation to generate optimized machine code for INT8 operations.
• Support multiple quantization schemes—per-tensor, per-channel, symmetric, and asymmetric—within the QNN dialect to preserve model accuracy while enabling efficient execution.
• Use the QNN dialect to decouple model quantization from hardware-specific optimizations, allowing a single compiler pipeline to target diverse platforms.
• Integrate the QNN representation into the compiler’s optimization pipeline to enable automatic fusion and layout transformations tailored for INT8 computation.

Experimental results

Research questions

• RQ1 Can a unified compiler-based approach efficiently execute pre-quantized INT8 models across diverse hardware platforms without requiring framework-specific porting?
• RQ2 How does the performance of a QNN-augmented deep learning compiler compare to native FP32 execution and framework-optimized INT8 execution on different hardware?
• RQ3 To what extent can a generic quantization context in a compiler dialect preserve model accuracy while enabling cross-platform deployment of quantized models?
• RQ4 Can the QNN dialect effectively support multiple quantization strategies (e.g., per-channel, symmetric) within a single compilation pipeline?

Key findings

• The QNN-augmented Apache TVM compiler achieves a 2.35× speedup on Intel Xeon Cascade Lake CPUs compared to optimized FP32 execution.
• On NVIDIA Tesla T4 GPUs, the QNN approach delivers a 2.15× speedup over FP32 execution, demonstrating strong GPU acceleration for INT8 inference.
• On ARM Cortex-A CPUs, the QNN compiler achieves 1.35× and 1.40× speedups on Raspberry Pi 3 and Pi 4, respectively, outperforming FP32 execution.
• The performance of the QNN-compiled models is comparable to state-of-the-art framework-specific solutions (e.g., TFLite, TensorRT) on their target platforms.
• The QNN approach enables cross-platform deployment of pre-quantized models with minimal developer effort, supporting models from TensorFlow Lite, PyTorch, and MXNet.
• The QNN dialect successfully decouples model quantization from hardware-specific optimizations, enabling a single toolchain to target diverse hardware platforms efficiently.