[Paper Review] Online Algorithms for Hierarchical Inference in Deep Learning applications at the Edge

Vishnu Narayanan Moothedath, Jaya Prakash Champati+1 more|arXiv (Cornell University)|Apr 3, 2023

Advanced Bandit Algorithms Research37 references4 citations

TL;DR

This paper proposes an online meta-learning framework for hierarchical inference in edge AI, where an edge device uses the maximum softmax value from a small ML model to decide whether to accept local inference or offload to a large model on an edge server. It introduces two algorithms—HIL-F for full feedback and HIL-N for no-local feedback—achieving sublinear regret bounds of $\sqrt{n\ln(1/\lambda_{\text{min}})/2}$ and $O(n^{2/3}\ln^{1/3}(1/\lambda_{\text{min}}))$, respectively, without requiring smoothness assumptions on the loss function.

ABSTRACT

We consider a resource-constrained Edge Device (ED), such as an IoT sensor or a microcontroller unit, embedded with a small-size ML model (S-ML) for a generic classification application and an Edge Server (ES) that hosts a large-size ML model (L-ML). Since the inference accuracy of S-ML is lower than that of the L-ML, offloading all the data samples to the ES results in high inference accuracy, but it defeats the purpose of embedding S-ML on the ED and deprives the benefits of reduced latency, bandwidth savings, and energy efficiency of doing local inference. In order to get the best out of both worlds, i.e., the benefits of doing inference on the ED and the benefits of doing inference on ES, we explore the idea of Hierarchical Inference (HI), wherein S-ML inference is only accepted when it is correct, otherwise the data sample is offloaded for L-ML inference. However, the ideal implementation of HI is infeasible as the correctness of the S-ML inference is not known to the ED. We propose an online meta-learning framework that the ED can use to predict the correctness of the S-ML inference. In particular, we propose to use the maximum softmax value output by S-ML for a data sample and decide whether to offload it or not. The resulting online learning problem turns out to be a Prediction with Expert Advice (PEA) problem with continuous expert space. We propose two different algorithms and prove sublinear regret bounds for them without any assumption on the smoothness of the loss function. We evaluate and benchmark the performance of the proposed algorithms for image classification application using four datasets, namely, Imagenette and Imagewoof, MNIST, and CIFAR-10.

Motivation & Objective

• To address the challenge of making real-time inference decisions on resource-constrained edge devices without knowing the correctness of small ML model predictions.
• To enable hierarchical inference that leverages local small ML models for efficiency while offloading uncertain predictions to larger models on edge servers for accuracy.
• To design an online learning framework that predicts the correctness of small ML inference using only the maximum softmax score, avoiding reliance on ground truth at inference time.
• To establish theoretical regret bounds for decision-making under uncertainty in both full feedback and no-local feedback scenarios.

Proposed method

• The framework uses the maximum softmax probability output by the small ML model as a confidence score to decide whether to accept local inference or offload to the large model.
• For the full feedback case (HIL-F), the algorithm treats the decision as a Prediction with Expert Advice problem with a continuous expert space, using exponential weighting over confidence thresholds.
• For the no-local feedback case (HIL-N), the algorithm employs a modified loss function and regret minimization with a pseudo-loss to handle the absence of ground truth during decision-making.
• Theoretical analysis derives regret bounds using online convex optimization and exponential weighting, with bounds independent of loss function smoothness.
• The algorithms are evaluated on image classification tasks using Imagenette, Imagewoof, MNIST, and CIFAR-10 datasets to validate performance and robustness.
• The framework is designed to be lightweight and suitable for deployment on microcontroller units and other resource-constrained edge devices.

Figure 1 : Schematic of the HI meta-learning framework

Experimental results

Research questions

• RQ1 Can an edge device reliably predict the correctness of its own small ML model inference without access to ground truth labels?
• RQ2 What is the optimal strategy for offloading uncertain predictions to a large model on an edge server while minimizing regret and preserving energy and bandwidth efficiency?
• RQ3 How can online learning be applied to a continuous expert space to make sequential decisions in hierarchical inference with minimal assumptions on loss function smoothness?
• RQ4 What regret bounds can be achieved in both full feedback and no-local feedback settings for this hierarchical inference problem?
• RQ5 How does the proposed framework compare to baseline strategies in terms of accuracy, latency, and energy efficiency across diverse image classification benchmarks?

Key findings

• The HIL-F algorithm achieves a regret bound of $\sqrt{n\ln(1/\lambda_{\text{min}})/2}$ in the full feedback setting, independent of the smoothness of the loss function.
• The HIL-N algorithm achieves a regret bound of $O(n^{2/3}\ln^{1/3}(1/\lambda_{\text{min}}))$ in the no-local feedback setting, demonstrating robustness without ground truth feedback.
• The regret bounds are derived without assuming smoothness of the loss function, making the framework applicable to a wide range of real-world edge inference scenarios.
• The framework maintains low computational overhead, making it suitable for deployment on microcontroller units and other resource-constrained edge devices.
• Empirical evaluation on four datasets shows that the proposed algorithms effectively balance accuracy, latency, and energy efficiency by minimizing unnecessary offloading.
• The theoretical analysis confirms that the regret remains bounded even when the large model is imperfect, with bounds scaling appropriately with offload cost and confidence threshold variability.

Figure 2 : Classification of Imagenette by a small-size quantized MobileNet using width multiplier $0.25$ [ 7 ] .