[Paper Review] A Unified Optimization Approach for CNN Model Inference on Integrated GPUs

Leyuan Wang, Zhi Chen+5 more|arXiv (Cornell University)|Jul 3, 2019

Advanced Neural Network Applications30 references4 citations

TL;DR

This paper presents a unified optimization framework for efficient CNN inference on integrated GPUs across diverse edge devices. It uses a domain-specific intermediate representation (IR) and machine learning-based scheduling to accelerate vision-specific operators, achieving up to 1.62× speedup over vendor-optimized libraries while supporting broader model coverage and fallback to CPU for compatibility.

ABSTRACT

Modern deep learning applications urge to push the model inference taking place at the edge devices for multiple reasons such as achieving shorter latency, relieving the burden of the network connecting to the cloud, and protecting user privacy. The Convolutional Neural Network (\emph{CNN}) is one of the most widely used model family in the applications. Given the high computational complexity of the CNN models, it is favorable to execute them on the integrated GPUs at the edge devices, which are ubiquitous and have more power and better energy efficiency than the accompanying CPUs. However, programming on integrated GPUs efficiently is challenging due to the variety of their architectures and programming interfaces. This paper proposes an end-to-end solution to execute CNN model inference on the integrated GPUs at the edge, which uses a unified IR to represent and optimize vision-specific operators on integrated GPUs from multiple vendors, as well as leverages machine learning-based scheduling search schemes to optimize computationally-intensive operators like convolution. Our solution even provides a fallback mechanism for operators not suitable or convenient to run on GPUs. The evaluation results suggest that compared to state-of-the-art solutions backed up by the vendor-provided high-performance libraries on Intel Graphics, ARM Mali GPU, and Nvidia integrated Maxwell GPU, our solution achieves similar, or even better (up to 1.62$ imes$), performance on a number of popular image classification and object detection models. In addition, our solution has a wider model coverage and is more flexible to embrace new models. Our solution has been adopted in production services in AWS and is open-sourced.

Motivation & Objective

• To address the challenge of inefficient and non-portable GPU inference on integrated GPUs in edge devices due to architectural heterogeneity and lack of portable optimization.
• To enable high-performance execution of vision-specific operators—previously under-optimized or offloaded to CPU—on integrated GPUs.
• To reduce developer burden by automating performance tuning across diverse SoC platforms without manual low-level kernel tuning.
• To provide a fallback mechanism for operators unsuitable for GPU execution, ensuring broad model compatibility.
• To achieve performance comparable or superior to vendor-optimized libraries while maintaining model portability and extensibility.

Proposed method

• A unified intermediate representation (IR) abstracts vision-specific operators and enables code generation for both CUDA and OpenCL across different GPU vendors.
• Machine learning-based scheduling is applied at both graph and tensor levels to search for optimal execution plans, reducing manual tuning effort.
• The framework includes specialized optimizations for control-intensive operators like NMS and ROIAlign, which are typically inefficient on GPUs.
• A fallback mechanism routes unsupported or suboptimal operators to the CPU on the same SoC, ensuring compatibility and performance.
• The system is built on a compiler-like pipeline inspired by Halide, extended to handle the unique characteristics of integrated GPUs.
• Performance tuning is automated using reinforcement learning or Bayesian optimization to explore efficient thread and memory layouts.

Experimental results

Research questions

• RQ1 How can vision-specific operators with complex control logic be efficiently executed on integrated GPUs across heterogeneous architectures?
• RQ2 Can a unified IR and automated scheduling strategy achieve performance comparable to vendor-optimized libraries on diverse integrated GPUs?
• RQ3 What is the impact of automated, ML-driven scheduling on inference latency and energy efficiency in edge CNN inference?
• RQ4 How can a unified framework support a broad range of popular CNN models while maintaining high performance across different SoC platforms?
• RQ5 To what extent can fallback to CPU improve model coverage and deployment flexibility without sacrificing overall performance?

Key findings

• The proposed framework achieves up to 1.62× speedup over vendor-optimized libraries (e.g., cuDNN, ACL, clDNN) on Intel, ARM, and NVIDIA integrated GPUs.
• Performance is comparable or better than state-of-the-art solutions across multiple image classification and object detection models, including ResNet, SSD, and YOLO.
• The framework supports a wider range of models than existing solutions, including those with vision-specific operators like NMS and ROIAlign that are often offloaded to CPU.
• The use of machine learning-based scheduling significantly reduces the need for manual tuning, enabling rapid deployment across diverse edge platforms.
• The fallback mechanism ensures that all operators can be executed efficiently, even when GPU acceleration is not feasible, enhancing model portability.
• The solution has been deployed in production at AWS and is open-sourced, demonstrating real-world viability and scalability.