[Paper Review] Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy

En Li, Zhi Zhou+1 more|arXiv (Cornell University)|Jun 20, 2018

Age of Information OptimizationComputer Science9 references52 citations

TL;DR

Edgent proposes a co-inference framework that jointly optimizes DNN partitioning between a mobile device and an edge server and DNN right-sizing with early-exits to meet latency deadlines while maximizing accuracy.

ABSTRACT

As the backbone technology of machine learning, deep neural networks (DNNs) have have quickly ascended to the spotlight. Running DNNs on resource-constrained mobile devices is, however, by no means trivial, since it incurs high performance and energy overhead. While offloading DNNs to the cloud for execution suffers unpredictable performance, due to the uncontrolled long wide-area network latency. To address these challenges, in this paper, we propose Edgent, a collaborative and on-demand DNN co-inference framework with device-edge synergy. Edgent pursues two design knobs: (1) DNN partitioning that adaptively partitions DNN computation between device and edge, in order to leverage hybrid computation resources in proximity for real-time DNN inference. (2) DNN right-sizing that accelerates DNN inference through early-exit at a proper intermediate DNN layer to further reduce the computation latency. The prototype implementation and extensive evaluations based on Raspberry Pi demonstrate Edgent's effectiveness in enabling on-demand low-latency edge intelligence.

Motivation & Objective

• Motivate low-latency DNN inference for mobile applications despite device and network constraints.
• Propose a co-inference framework that leverages device-edge synergy for DNN execution.
• Introduce adaptive DNN partitioning and early-exit right-sizing to meet predefined latency deadlines.
• Provide an offline-online workflow to predict per-layer latency and optimize partition/exit points.
• Demonstrate feasibility with a Raspberry Pi-based prototype and empirical evaluations.

Proposed method

• Partitioning: adaptively split DNN computation between device and edge to minimize latency under bandwidth constraints.
• Right-sizing: enable early-exits in DNNs to reduce computation and enable latency-accuracy tradeoffs.
• Offline profiling: build regression-based models predicting per-layer latency on device and edge and train branchy networks with multiple exits.
• Online optimization: jointly optimize exit point and partition point to maximize accuracy subject to a latency deadline using bandwidth and latency inputs.
• Co-inference: execute pre-partitioned layers on the edge and remaining layers on the device according to the selected plan.

Experimental results

Research questions

• RQ1 How can DNN partitioning between device and edge reduce end-to-end latency under varying bandwidth?
• RQ2 Can DNN right-sizing via early exits improve latency while preserving accuracy within a deadline?
• RQ3 What is the optimal combination of partition point and exit point to maximize accuracy under a given latency constraint?
• RQ4 How effective is a regression-based latency predictor for guiding online optimization across different layer types?
• RQ5 Is the proposed offline-online Edgent workflow feasible on commodity hardware for real-time edge intelligence?

Key findings

• Latency can be reduced by jointly optimizing partition and exit points compared to device-only or edge-only execution.
• Regression-based per-layer latency models enable fast online optimization (≤1 ms in experiments).
• Higher bandwidth allows selecting higher-accuracy exit models, improving end-to-end accuracy within the latency constraint.
• The best exit point generally increases as bandwidth grows or latency requirements relax.
• Prototype experiments show Edgent can meet hard latency goals and outpace baseline approaches under varying bandwidths.