[Paper Review] HG-Caffe: Mobile and Embedded Neural Network GPU (OpenCL) Inference Engine with FP16 Supporting

Zhuoran Ji|arXiv (Cornell University)|Jan 3, 2019

Advanced Neural Network Applications13 references4 citations

TL;DR

HG-Caffe is a GPU-accelerated, OpenCL-based deep learning inference engine for mobile and embedded devices that supports half-precision (FP16) computation. It achieves up to 20× speedup over CPU inference and reduces peak memory usage to 80% of the original Caffe, enabling efficient, low-power deployment of deep neural networks on edge AI platforms with Mali and Adreno GPUs.

ABSTRACT

Breakthroughs in the fields of deep learning and mobile system-on-chips are radically changing the way we use our smartphones. However, deep neural networks inference is still a challenging task for edge AI devices due to the computational overhead on mobile CPUs and a severe drain on the batteries. In this paper, we present a deep neural network inference engine named HG-Caffe, which supports GPUs with half precision. HG-Caffe provides up to 20 times speedup with GPUs compared to the original implementations. In addition to the speedup, the peak memory usage is also reduced to about 80%. With HG-Caffe, more innovative and fascinating mobile applications will be turned into reality.

Motivation & Objective

• To address the high computational overhead and battery drain caused by CPU-based deep neural network inference on mobile and embedded devices.
• To enable efficient, general-purpose deep learning inference on mobile GPUs using OpenCL, which is widely supported across mobile SoC architectures.
• To support half-precision (FP16) arithmetic across all neural network layers to further improve throughput and reduce memory usage.
• To maintain compatibility with existing BVLC Caffe models by preserving the same weight file format and layer definitions.
• To reduce software bloat and dependencies by removing third-party libraries, enhancing portability and maintainability for edge deployment.

Proposed method

• HG-Caffe is built as a fork of BVLC Caffe with third-party library dependencies removed to improve maintainability and portability.
• It leverages OpenCL to enable GPU execution across diverse mobile GPU architectures, including Mali and Adreno, ensuring broad hardware compatibility.
• The framework implements full FP16 support across all layers, reducing memory bandwidth pressure and arithmetic cost while maintaining model accuracy.
• It operates in forward-only mode, significantly reducing peak memory usage and software footprint compared to full training frameworks.
• A Caffe weight file converter is included to enable direct use of pre-trained models from BVLC Caffe without retraining.
• Optimizations are applied to kernel launches and memory access patterns to maximize GPU utilization and throughput on mobile GPUs.

Experimental results

Research questions

• RQ1 Can a general-purpose deep learning inference engine achieve significant speedup on mobile GPUs compared to CPU-only execution?
• RQ2 To what extent does FP16 arithmetic improve inference throughput and reduce memory usage on mobile GPUs?
• RQ3 How does the performance of GPU-based inference vary across different mobile SoCs, such as Snapdragon 845 and Kirin 970, under varying batch sizes?
• RQ4 Can a GPU-accelerated, FP16-capable inference engine maintain compatibility with existing BVLC Caffe models and weight formats?
• RQ5 What is the impact of memory constraints on GPU performance in mobile devices, particularly when using large models?

Key findings

• On Snapdragon 845 SoCs, GPU (FP32) inference achieves up to 20× speedup over CPU (FP32), with FP16 providing an additional 2× throughput improvement.
• The peak memory usage of GPU (FP32) is reduced to 29.1% of CPU (FP32), and GPU (FP16) further reduces it to 20.8% of CPU (FP32).
• With batch size 1, GPU (FP32) is slower than CPU (FP32) in Style Transfer Net, but performance improves significantly with larger batch sizes.
• On Kirin 970 SoCs, GPU performance is limited by a 100 MB GPU memory ceiling, causing frequent memory swapping and poor performance for deep networks.
• The framework reduces peak memory usage to 80% of the original BVLC Caffe, enhancing efficiency on memory-constrained edge devices.
• HG-Caffe maintains full compatibility with BVLC Caffe’s model format, enabling direct deployment of pre-trained models without retraining.