[Paper Review] Auto-tuning Neural Network Quantization Framework for Collaborative Inference Between the Cloud and Edge

Guangli Li, Lei Liu+4 more|arXiv (Cornell University)|Dec 16, 2018

Advanced Neural Network Applications11 references4 citations

TL;DR

This paper proposes an auto-tuning neural network quantization framework for cloud-edge collaborative inference that automatically identifies optimal mixed-precision partition points in deep neural networks. By quantizing early layers for execution on mobile edge devices and offloading later layers to the cloud, the framework reduces model storage by up to 99.97% and achieves 1.7× speedup with less than 0.2% accuracy loss on ImageNet benchmarks.

ABSTRACT

Recently, deep neural networks (DNNs) have been widely applied in mobile intelligent applications. The inference for the DNNs is usually performed in the cloud. However, it leads to a large overhead of transmitting data via wireless network. In this paper, we demonstrate the advantages of the cloud-edge collaborative inference with quantization. By analyzing the characteristics of layers in DNNs, an auto-tuning neural network quantization framework for collaborative inference is proposed. We study the effectiveness of mixed-precision collaborative inference of state-of-the-art DNNs by using ImageNet dataset. The experimental results show that our framework can generate reasonable network partitions and reduce the storage on mobile devices with trivial loss of accuracy.

Motivation & Objective

• To address high transmission overhead and privacy risks in cloud-only DNN inference on mobile devices.
• To reduce model storage and communication costs in mobile AI applications through edge deployment.
• To enable efficient, privacy-preserving collaborative inference using mixed-precision quantization.
• To automate the selection of optimal network partition points for cloud-edge workloads.
• To minimize accuracy degradation while maximizing inference speed and storage reduction.

Proposed method

• Analyzes DNN layer structures to identify candidate partition points based on computational and memory characteristics.
• Applies INT8 scalar quantization to early network layers for reduced storage and computation on edge devices.
• Uses on-device profiling to measure inference latency and communication overhead for each candidate partition.
• Employs an auto-tuning algorithm to evaluate all candidate partitions and select the optimal one based on speed and accuracy trade-offs.
• Executes the first part of the network on the edge (quantized) and the remainder in the cloud (full-precision) to balance performance and privacy.
• Leverages optimized low-precision libraries (e.g., gemmlowp, cuDNN) to accelerate inference on both edge and cloud.

Experimental results

Research questions

• RQ1 Which layers in a DNN are most suitable for partitioning in a cloud-edge collaborative inference setup?
• RQ2 How can mixed-precision quantization be used to minimize model size and communication overhead without significant accuracy loss?
• RQ3 What is the optimal network partition point that balances inference speed, storage, and accuracy in wireless environments?
• RQ4 Can auto-tuning effectively identify the best partition for diverse state-of-the-art DNNs like ResNet and VGG?
• RQ5 To what extent does collaborative inference reduce data transmission and improve privacy compared to cloud-only inference?

Key findings

• The framework achieved a 1.7× speedup for AlexNet by selecting conv5 as the optimal partition point, reducing transmission overhead in low-bandwidth environments.
• Model storage on mobile devices was reduced by 96.17% for AlexNet, 99.97% for VGG16, 85.63% for ResNet-18, and 98.22% for GoogLeNet.
• The largest accuracy drop across all tested networks was only -0.11%, with most models showing negligible degradation (≤0.1%).
• Collaborative inference outperformed cloud-only inference in speed when transmission overhead was high, especially in low-bandwidth wireless conditions.
• The framework successfully identified the best and fastest partition points for each network, demonstrating robustness across diverse architectures.
• Intermediate feature maps were transmitted in low-precision format, reducing communication volume and enhancing user privacy.