[Paper Review] Computation Offloading in Beyond 5G Networks: A Distributed Learning Framework and Applications

Xianfu Chen, Celimuge Wu+3 more|arXiv (Cornell University)|Jul 15, 2020

IoT and Edge/Fog Computing13 references4 citations

TL;DR

This paper proposes a distributed reinforcement learning framework for computation offloading in beyond 5G networks, modeling the problem as a multi-agent Markov decision process to enable autonomous, scalable decision-making. The proposed online deep RL algorithm achieves superior average utility performance compared to benchmark channel- and queue-aware strategies, with convergence observed in approximately 13,000 time steps under dynamic traffic loads.

ABSTRACT

Facing the trend of merging wireless communications and multi-access edge computing (MEC), this article studies computation offloading in the beyond fifth-generation networks. To address the technical challenges originating from the uncertainties and the sharing of limited resource in an MEC system, we formulate the computation offloading problem as a multi-agent Markov decision process, for which a distributed learning framework is proposed. We present a case study on resource orchestration in computation offloading to showcase the potentials of an online distributed reinforcement learning algorithm developed under the proposed framework. Experimental results demonstrate that our learning algorithm outperforms the benchmark resource orchestration algorithms. Furthermore, we outline the research directions worth in-depth investigation to minimize the time cost, which is one of the main practical issues that prevent the implementation of the proposed distributed learning framework.

Motivation & Objective

• To address the challenges of dynamic resource sharing and uncertainty in beyond 5G multi-access edge computing (MEC) systems.
• To enable scalable, autonomous computation offloading decisions across multiple mobile terminals (MTs) without centralized coordination.
• To develop a distributed learning framework that supports real-time adaptation to changing network states and workloads.
• To minimize time cost in training while maintaining high performance in resource orchestration.

Proposed method

• Formulate computation offloading as a multi-agent Markov decision process (MADP) to model interdependent decisions among MTs.
• Design a distributed learning framework where each MT acts as an agent using only local observations and contextual information.
• Implement an online deep reinforcement learning algorithm to dynamically optimize offloading decisions based on real-time feedback (rewards).
• Integrate supervised learning components to improve policy abstraction and reduce training instability.
• Utilize experience replay and target networks to stabilize training in the deep RL component.
• Apply function approximation via deep neural networks to handle high-dimensional state-action spaces in real-time.

Experimental results

Research questions

• RQ1 How can computation offloading be optimized in beyond 5G networks under dynamic, shared resource constraints?
• RQ2 What is the impact of decentralized decision-making on system scalability and performance in multi-agent MEC environments?
• RQ3 Can online distributed reinforcement learning achieve better utility than centralized or heuristic benchmarks in real-time offloading?
• RQ4 How does the convergence time of online learning affect practical deployment in large-scale MEC systems?
• RQ5 What techniques can accelerate training and reduce time cost in distributed offloading frameworks?

Key findings

• The proposed online deep RL algorithm converges within approximately 13,000 time steps during the training process.
• The algorithm achieves significantly better average utility performance than both the Channel-Aware and Queue-Aware benchmark algorithms.
• As traffic load increases, the algorithm reduces packet drops more effectively than the Channel-Aware approach, despite higher local CPU energy consumption.
• The algorithm maintains lower average queue lengths and fewer packet drops compared to the Channel-Aware baseline under varying data arrival rates.
• The framework enables autonomous, scalable decision-making without requiring inter-agent communication or centralized coordination.
• The study identifies off-policy learning and transfer learning as key enablers for reducing training time and accelerating convergence in practical deployments.