[Paper Review] Reinforcement Learning Based Orchestration for Elastic Services

Mauricio Fadel Argerich, Bin Cheng+1 more|arXiv (Cornell University)|Apr 26, 2019

IoT and Edge/Fog Computing9 references4 citations

TL;DR

This paper proposes a reinforcement learning (RL)-based orchestration framework for elastic services in edge computing environments to dynamically adapt service behavior to varying execution contexts. By learning optimal configuration policies through trial-and-error interaction, the approach achieves 10–25% higher precision and 25% lower computational overhead than heuristic-based methods, with under 0.5% runtime overhead on the service itself.

ABSTRACT

Due to the highly variable execution context in which edge services run, adapting their behavior to the execution context is crucial to comply with their requirements. However, adapting service behavior is a challenging task because it is hard to anticipate the execution contexts in which it will be deployed, as well as assessing the impact that each behavior change will produce. In order to provide this adaptation efficiently, we propose a Reinforcement Learning (RL) based Orchestration for Elastic Services. We implement and evaluate this approach by adapting an elastic service in different simulated execution contexts and comparing its performance to a Heuristics based approach. We show that elastic services achieve high precision and requirement satisfaction rates while creating an overhead of less than 0.5% to the overall service. In particular, the RL approach proves to be more efficient than its rule-based counterpart; yielding a 10 to 25% higher precision while being 25% less computationally expensive.

Motivation & Objective

• To address the challenge of dynamically adapting service behavior in heterogeneous, resource-constrained edge environments with unpredictable workloads.
• To reduce the reliance on pre-defined heuristics that may not generalize across diverse execution contexts.
• To enable self-adaptive service orchestration that learns optimal configurations through experience, improving both precision and latency satisfaction.
• To evaluate the performance of RL-based orchestration against heuristic-based approaches in simulated edge scenarios.
• To demonstrate that RL-based orchestration achieves high requirement satisfaction with minimal runtime overhead.

Proposed method

• The approach models service adaptation as a Markov Decision Process (MDP), defining states as system and workload conditions, actions as configuration parameter changes, and rewards as a function of precision and latency satisfaction.
• A tabular Q-learning algorithm is used to train the agent to select actions that maximize cumulative reward based on observed states.
• The state space includes CPU availability, number of faces per frame, and network conditions; actions include adjusting image resolution, preprocessing steps, and model inference settings.
• The reward function balances high precision (accurate detection) with low latency (meeting SLOs), penalizing violations of end-to-end response time constraints.
• The system is evaluated in simulation using real-world video analysis workloads, including the Lost Child Application, under varying load and hardware conditions.
• Performance is compared against a heuristic-based orchestration strategy using fixed rules for configuration changes.

Experimental results

Research questions

• RQ1 Can reinforcement learning effectively learn optimal configuration policies for elastic services in dynamic edge environments?
• RQ2 How does RL-based orchestration compare to heuristic-based approaches in terms of precision and latency satisfaction?
• RQ3 What is the runtime overhead of the RL-based orchestration mechanism relative to the overall service execution?
• RQ4 How quickly can the RL agent adapt to sudden changes in system resources, such as CPU availability?
• RQ5 Can the RL approach generalize across diverse workloads and hardware configurations without manual reconfiguration?

Key findings

• The RL-based orchestration achieved 10–25% higher precision than the heuristic-based approach across all tested workloads, including random, full-day, and variable load patterns.
• The RL approach maintained a latency satisfaction rate of 94.72% under high load, outperforming the heuristic’s 79.06% in the same scenario.
• The RL agent reduced computational overhead to 0.30% on Intel Core i5 and 0.30% on Raspberry Pi 3 B+, compared to 0.42% for the heuristic approach.
• The RL-based system adapted within 30 steps to a drop in CPU availability at step 485, reducing precision slightly but maintaining latency compliance.
• When CPU availability recovered at step 515, the system resumed higher-precision configurations, demonstrating rapid and effective adaptation.
• The RL-based approach consumed 25% less execution time than the heuristic method, indicating improved computational efficiency.