[Paper Review] Online Service Migration in Mobile Edge with Incomplete System Information: A Deep Recurrent Actor-Critic Learning Approach

Jin Wang, Jia Hu+3 more|arXiv (Cornell University)|Dec 16, 2020

IoT and Edge/Fog Computing29 references4 citations

TL;DR

This paper proposes DRACM, a user-centric deep reinforcement learning method for online service migration in mobile edge computing under incomplete system information. By modeling the problem as a Partially Observable Markov Decision Process (POMDP) and using a novel LSTM-embedding encoder with a clipped surrogate off-policy actor-critic algorithm, DRACM achieves stable training and near-optimal performance across diverse real-world mobility scenarios.

ABSTRACT

Multi-access Edge Computing (MEC) is an emerging computing paradigm that extends cloud computing to the network edge to support resource-intensive applications on mobile devices. As a crucial problem in MEC, service migration needs to decide how to migrate user services for maintaining the Quality-of-Service when users roam between MEC servers with limited coverage and capacity. However, finding an optimal migration policy is intractable due to the dynamic MEC environment and user mobility. Many existing studies make centralized migration decisions based on complete system-level information, which is time-consuming and also lacks desirable scalability. To address these challenges, we propose a novel learning-driven method, which is user-centric and can make effective online migration decisions by utilizing incomplete system-level information. Specifically, the service migration problem is modeled as a Partially Observable Markov Decision Process (POMDP). To solve the POMDP, we design a new encoder network that combines a Long Short-Term Memory (LSTM) and an embedding matrix for effective extraction of hidden information, and further propose a tailored off-policy actor-critic algorithm for efficient training. The extensive experimental results based on real-world mobility traces demonstrate that this new method consistently outperforms both the heuristic and state-of-the-art learning-driven algorithms and can achieve near-optimal results on various MEC scenarios.

Motivation & Objective

• Address the challenge of online service migration in Multi-access Edge Computing (MEC) under incomplete system-level information.
• Overcome the scalability and latency limitations of centralized, information-complete migration approaches in dynamic, mobile environments.
• Develop a decentralized, model-free learning method that enables mobile users to make effective migration decisions autonomously with minimal human expertise.
• Stabilize training and improve performance in complex MEC environments with continuous state spaces and dynamic user mobility.
• Enable real-time, online migration decisions through offline policy training while minimizing direct interaction with the live MEC environment.

Proposed method

• Formulate the service migration problem as a Partially Observable Markov Decision Process (POMDP) to capture complex system dynamics and incomplete information.
• Design a hybrid encoder network combining Long Short-Term Memory (LSTM) and an embedding matrix to extract hidden state representations from historical, partial observations.
• Propose a tailored off-policy actor-critic reinforcement learning algorithm with a clipped surrogate objective function to stabilize training and improve sample efficiency.
• Decouple policy training (offline) from inference (online), allowing real-time migration decisions without direct interaction with the live MEC system.
• Use end-to-end deep reinforcement learning to learn migration policies directly from raw environmental observations, minimizing reliance on handcrafted heuristics.
• Leverage real-world mobility traces for training and evaluation, ensuring the method generalizes across diverse mobility patterns and MEC deployment scenarios.

Experimental results

Research questions

• RQ1 Can a model-free, user-centric deep reinforcement learning approach effectively handle online service migration in MEC with incomplete system information?
• RQ2 How does the proposed DRACM method perform compared to heuristic and state-of-the-art learning-based baselines in terms of QoS and migration cost?
• RQ3 To what extent does the integration of LSTM and embedding layers improve the representation of partial historical observations in the POMDP setting?
• RQ4 Does the use of a clipped surrogate objective in the off-policy actor-critic framework lead to more stable and efficient training than existing Q-learning-based methods in continuous-state MEC environments?
• RQ5 Can the DRACM framework achieve near-optimal performance while enabling online decision-making with minimal interaction with the live MEC system?

Key findings

• DRACM consistently outperforms both heuristic and state-of-the-art learning-driven baselines in terms of service migration cost and QoS maintenance across diverse real-world mobility scenarios.
• The method achieves near-optimal performance, demonstrating strong generalization across different MEC deployment configurations and user mobility patterns.
• The use of an LSTM-embedding encoder enables effective extraction of hidden temporal dependencies from incomplete observation histories, improving decision-making accuracy.
• The clipped surrogate objective in the off-policy actor-critic algorithm leads to significantly more stable training compared to independent Q-learning baselines.
• Offline training of the policy network enables online inference with low latency, making the method practical for real-time deployment on mobile devices.
• The framework successfully decouples training from online operation, reducing the cost and risk of direct interaction with the live MEC environment.