[Paper Review] Deep Reinforcement Learning for Online Offloading in Wireless Powered Mobile-Edge Computing Networks.

Liang Huang, Suzhi Bi+1 more|arXiv (Cornell University)|Aug 6, 2018

Energy Harvesting in Wireless Networks25 citations

TL;DR

This paper proposes DROO, a deep reinforcement learning framework that enables real-time, online task offloading in wireless-powered mobile-edge computing networks by learning optimal binary offloading decisions and resource allocations. It achieves near-optimal performance with over 10× lower computation latency than conventional methods, reducing CPU execution time to under 0.1 seconds in a 30-user network.

ABSTRACT

Wireless powered mobile-edge computing (MEC) has recently emerged as a promising paradigm to enhance the data processing capability of low-power networks, such as wireless sensor networks and internet of things (IoT). In this paper, we consider a wireless powered MEC network that adopts a binary offloading policy, so that each computation task of wireless devices (WDs) is either executed locally or fully offloaded to an MEC server. Our goal is to acquire an online algorithm that optimally adapts task offloading decisions and wireless resource allocations to the time-varying wireless channel conditions. This requires quickly solving hard combinatorial optimization problems within the channel coherence time, which is hardly achievable with conventional numerical optimization methods. To tackle this problem, we propose a Deep Reinforcement learning-based Online Offloading (DROO) framework that implements a deep neural network as a scalable solution that learns the binary offloading decisions from the experience. It eliminates the need of solving combinatorial optimization problems, and thus greatly reduces the computational complexity especially in large-size networks. To further reduce the complexity, we propose an adaptive procedure that automatically adjusts the parameters of the DROO algorithm on the fly. Numerical results show that the proposed algorithm can achieve near-optimal performance while significantly decreasing the computation time by more than an order of magnitude compared with existing optimization methods. For example, the CPU execution latency of DROO is less than $0.1$ second in a $30$-user network, making real-time and optimal offloading truly viable even in a fast fading environment.

Motivation & Objective

• Address the challenge of online task offloading in wireless-powered mobile-edge computing (MEC) networks with time-varying channel conditions.
• Overcome the high computational complexity of solving hard combinatorial optimization problems within the channel coherence time using conventional methods.
• Design a scalable, real-time solution that dynamically adapts offloading decisions and resource allocation without solving optimization problems online.
• Reduce computational latency significantly to enable practical deployment in fast-fading environments, especially in large-scale IoT and sensor networks.

Proposed method

• Proposes a Deep Reinforcement Learning-based Online Offloading (DROO) framework using a deep neural network to learn binary offloading decisions from experience.
• Models the offloading decision as a Markov decision process, where the agent selects actions (local execution or full offloading) based on current channel state and system state.
• Employs a Q-learning-inspired deep Q-network (DQN) architecture to approximate the action-value function and enable end-to-end learning of optimal policies.
• Introduces an adaptive parameter adjustment procedure that dynamically tunes hyperparameters of the DROO algorithm during runtime to improve convergence and performance.
• Uses a reward function that jointly optimizes energy efficiency and task completion delay, reflecting the trade-off between local and offloaded computation.
• Trains the DROO agent in a simulated environment with time-varying channel conditions to generalize across diverse network states and user counts.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning framework effectively learn optimal online offloading decisions in wireless-powered MEC networks with time-varying channels?
• RQ2 How does the DROO framework compare to conventional optimization methods in terms of computation latency and performance scalability?
• RQ3 To what extent can adaptive hyperparameter tuning improve the convergence and robustness of the DROO algorithm in dynamic environments?
• RQ4 What is the achievable trade-off between energy efficiency and task completion delay in the proposed online offloading framework?
• RQ5 Can the DROO framework maintain near-optimal performance while reducing computational complexity by more than an order of magnitude in large-scale networks?

Key findings

• The DROO framework achieves near-optimal performance in terms of energy-delay trade-off, closely approaching the theoretical optimum obtained via exhaustive search.
• The CPU execution latency of DROO is less than 0.1 seconds in a 30-user network, demonstrating real-time feasibility even in fast-fading environments.
• DROO reduces computation time by more than an order of magnitude compared to conventional numerical optimization methods, significantly improving scalability.
• The adaptive parameter adjustment procedure enhances learning stability and convergence speed, especially in high-dynamic channel conditions.
• The framework maintains high performance across various network sizes, with minimal performance degradation in large-scale deployments.
• The deep neural network component of DROO successfully generalizes across diverse channel states and user distributions without requiring retraining for each scenario.