[Paper Review] Multi-user Resource Control with Deep Reinforcement Learning in IoT Edge Computing

Lei Lei, Huijuan Xu+4 more|arXiv (Cornell University)|Jun 19, 2019

IoT and Edge/Fog Computing36 references4 citations

TL;DR

This paper proposes a deep reinforcement learning-based joint computation offloading and multi-user scheduling algorithm for IoT edge computing systems to minimize long-term average weighted sum of delay and power consumption. Using a continuous-time Markov decision process (CTMDP) model and a novel neural network architecture for value function approximation, the method enables semi-distributed auction-based resource control that effectively mitigates the curse of dimensionality, achieving significant performance gains over baseline and alternative DRL methods in simulations.

ABSTRACT

By leveraging the concept of mobile edge computing (MEC), massive amount of data generated by a large number of Internet of Things (IoT) devices could be offloaded to MEC server at the edge of wireless network for further computational intensive processing. However, due to the resource constraint of IoT devices and wireless network, both the communications and computation resources need to be allocated and scheduled efficiently for better system performance. In this paper, we propose a joint computation offloading and multi-user scheduling algorithm for IoT edge computing system to minimize the long-term average weighted sum of delay and power consumption under stochastic traffic arrival. We formulate the dynamic optimization problem as an infinite-horizon average-reward continuous-time Markov decision process (CTMDP) model. One critical challenge in solving this MDP problem for the multi-user resource control is the curse-of-dimensionality problem, where the state space of the MDP model and the computation complexity increase exponentially with the growing number of users or IoT devices. In order to overcome this challenge, we use the deep reinforcement learning (RL) techniques and propose a neural network architecture to approximate the value functions for the post-decision system states. The designed algorithm to solve the CTMDP problem supports semi-distributed auction-based implementation, where the IoT devices submit bids to the BS to make the resource control decisions centrally. Simulation results show that the proposed algorithm provides significant performance improvement over the baseline algorithms, and also outperforms the RL algorithms based on other neural network architectures.

Motivation & Objective

• To address the challenge of efficient resource allocation in multi-user IoT edge computing systems with constrained computation and wireless resources.
• To minimize the long-term average weighted sum of delay and power consumption under stochastic traffic arrivals.
• To overcome the curse-of-dimensionality in multi-user Markov decision processes (MDPs) through scalable deep reinforcement learning.
• To enable semi-distributed implementation via auction-based resource control at the base station.
• To design a neural network architecture that accurately approximates value functions for post-decision states in continuous-time MDPs.

Proposed method

• Formulates the dynamic resource control problem as an infinite-horizon average-reward continuous-time Markov decision process (CTMDP).
• Introduces a neural network architecture with a single convolutional layer to approximate value functions for post-decision system states, reducing computational complexity.
• Employs a dual learning rate scheme: one for value function updates and another for average reward estimation, ensuring convergence.
• Uses a semi-distributed auction mechanism where IoT devices submit bids to the base station for resource allocation decisions.
• Applies temporal difference learning with eligibility traces, updating per-node value functions and weight vectors using state-dependent gradients.
• Implements a function approximation framework with separate updates for convolutional and fully connected layer weights, using the derivative of the Tanh activation function.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning effectively solve the curse-of-dimensionality problem in multi-user IoT edge computing resource control?
• RQ2 How does the proposed neural network architecture for value function approximation compare to other DRL architectures in terms of performance and convergence?
• RQ3 What is the impact of semi-distributed auction-based implementation on system scalability and fairness in multi-user scenarios?
• RQ4 To what extent does the algorithm reduce the long-term weighted sum of delay and power consumption compared to baseline methods?
• RQ5 How do different learning rate sequences affect the convergence and stability of the proposed algorithm?

Key findings

• The proposed Neural-ICO algorithm achieves significant performance improvement over baseline algorithms in minimizing the long-term average weighted sum of delay and power consumption.
• The algorithm outperforms other deep reinforcement learning methods based on different neural network architectures, particularly in high-dimensional multi-user environments.
• Simulation results demonstrate stable convergence and robustness under varying traffic arrival patterns and user loads.
• The use of a single convolutional layer in the function approximation architecture yields optimal performance, with additional fully connected layers providing no benefit.
• The dual learning rate scheme with faster-decaying αk for average reward estimation ensures accurate long-term reward tracking and stable policy learning.
• The semi-distributed auction-based implementation enables scalable and practical deployment in real-world IoT edge networks.