[Paper Review] Offloading Algorithms for Maximizing Inference Accuracy on Edge Device Under a Time Constraint

Andrea Fresa, Jaya Prakash Champati|arXiv (Cornell University)|Dec 21, 2021

IoT and Edge/Fog Computing4 citations

TL;DR

This paper proposes AMR², an approximation algorithm for offloading machine learning inference jobs between edge devices (EDs) and edge servers (ESs) to maximize accuracy under a time constraint T. By using LP relaxation and rounding, AMR² ensures makespan ≤ 2T and achieves near-optimal accuracy, with a 40% average improvement over greedy baselines in experiments on Raspberry Pi with MobileNet and ResNet models.

ABSTRACT

With the emergence of edge computing, the problem of offloading jobs between an Edge Device (ED) and an Edge Server (ES) received significant attention in the past. Motivated by the fact that an increasing number of applications are using Machine Learning (ML) inference, we study the problem of offloading inference jobs by considering the following novel aspects: 1) in contrast to a typical computational job, the processing time of an inference job depends on the size of the ML model, and 2) recently proposed Deep Neural Networks (DNNs) for resource-constrained devices provide the choice of scaling the model size. We formulate an assignment problem with the aim of maximizing the total inference accuracy of n data samples available at the ED, subject to a time constraint T on the makespan. We propose an approximation algorithm AMR2, and prove that it results in a makespan at most 2T, and achieves a total accuracy that is lower by a small constant from optimal total accuracy. As proof of concept, we implemented AMR2 on a Raspberry Pi, equipped with MobileNet, and is connected to a server equipped with ResNet, and studied the total accuracy and makespan performance of AMR2 for image classification application.

Motivation & Objective

• To address the challenge of offloading ML inference jobs between edge devices and edge servers while balancing inference accuracy and execution time.
• To model the trade-off between model size (affecting processing time and accuracy) and communication overhead in edge computing.
• To formulate a novel general assignment problem that maximizes total inference accuracy under a makespan constraint T.
• To design an efficient approximation algorithm (AMR²) with bounded makespan and near-optimal accuracy for heterogeneous, size-scaled ML models on EDs.
• To provide an optimal dynamic programming solution (AMDP) when all data samples are identical.

Proposed method

• Formulates the offloading problem as an Integer Linear Program (ILP) to maximize total inference accuracy under a time constraint T.
• Applies LP relaxation to the ILP to obtain a fractional solution, followed by a rounding procedure to assign jobs to ED or ES.
• Uses a two-phase rounding strategy: first assign fractional jobs to the ES, then reassign any resulting makespan violations via a greedy reassignment step.
• Proves that AMR² guarantees a makespan of at most 2T and a total accuracy loss of at most 2 compared to the optimal solution.
• For identical jobs, proposes AMDP, a pseudo-polynomial time dynamic programming algorithm that computes the optimal schedule.
• Empirically evaluates AMR² on a Raspberry Pi with MobileNet (ED) and ResNet (ES), using real processing and communication times from image classification workloads.

Experimental results

Research questions

• RQ1 How can inference accuracy be maximized under a strict time constraint when edge devices have multiple model sizes with varying accuracy and processing times?
• RQ2 What is the theoretical performance bound of an approximation algorithm for offloading inference jobs with variable model sizes and communication delays?
• RQ3 How does the proposed AMR² algorithm compare to greedy baselines in terms of accuracy and makespan in real-world edge deployments?
• RQ4 Under what conditions does the LP-relaxation-based approach yield near-optimal accuracy despite rounding-induced deviations?
• RQ5 Can an optimal solution be computed efficiently when all data samples are identical, and what is the time complexity of such a solution?

Key findings

• AMR² guarantees a makespan of at most 2T, ensuring feasibility within a relaxed time bound.
• The total inference accuracy of AMR² is at most 2 lower than the optimal solution, and in typical cases, the loss is bounded by the difference in accuracy between the largest and smallest model (a_{m+1} - a_1).
• In real-world experiments on Raspberry Pi, AMR² achieved an average of 40% higher total inference accuracy than the Greedy-RRA baseline.
• For T = 0.5 seconds, AMR² achieved a 20–60% accuracy gain over Greedy-RRA, with lower gains at smaller T due to limited offloading opportunities.
• For T = 4 seconds, AMR² achieved 40–50% higher accuracy than Greedy-RRA, demonstrating scalability under larger time budgets.
• The real-time makespan measured on Raspberry Pi closely matched the estimated makespan, confirming the accuracy of the timing model used in the algorithm.