[Paper Review] Multi-agent Reinforcement Learning for Resource Allocation in IoT networks with Edge Computing

Xiaolan Liu, Jiadong Yu+1 more|arXiv (Cornell University)|Apr 5, 2020

IoT and Edge/Fog Computing28 references4 citations

TL;DR

This paper proposes an independent learners-based multi-agent Q-learning (IL-based MA-Q) algorithm for distributed computation offloading in IoT edge computing networks, where each end user autonomously learns optimal decisions on local vs. edge execution, radio access technology, and sub-channel selection to minimize long-term system cost. The method achieves lower average cost and improved energy efficiency than centralized and random benchmarks, with reduced gateway overhead and no need for centralized channel estimation.

ABSTRACT

To support popular Internet of Things (IoT) applications such as virtual reality, mobile games and wearable devices, edge computing provides a front-end distributed computing archetype of centralized cloud computing with low latency. However, it's challenging for end users to offload computation due to their massive requirements on spectrum and computation resources and frequent requests on Radio Access Technology (RAT). In this paper, we investigate computation offloading mechanism with resource allocation in IoT edge computing networks by formulating it as a stochastic game. Here, each end user is a learning agent observing its local environment to learn optimal decisions on either local computing or edge computing with the goal of minimizing long term system cost by choosing its transmit power level, RAT and sub-channel without knowing any information of the other end users. Therefore, a multi-agent reinforcement learning framework is developed to solve the stochastic game with a proposed independent learners based multi-agent Q-learning (IL-based MA-Q) algorithm. Simulations demonstrate that the proposed IL-based MA-Q algorithm is feasible to solve the formulated problem and is more energy efficient without extra cost on channel estimation at the centralized gateway compared to the other two benchmark algorithms.

Motivation & Objective

• To address the challenge of resource-constrained, dynamic IoT edge networks with high demands on spectrum, computation, and frequent Radio Access Technology (RAT) switching.
• To enable end users to make autonomous, optimal offloading decisions without prior knowledge of other users’ actions.
• To reduce the computational and signaling burden on the centralized gateway by decentralizing decision-making.
• To minimize long-term system cost through joint optimization of transmit power, RAT, and sub-channel allocation.

Proposed method

• Formulates the computation offloading problem as a stochastic game where each end user acts as an independent learning agent.
• Develops an IL-based MA-Q algorithm enabling each agent to learn its optimal policy via Q-learning in a decentralized manner.
• Each agent maintains its own Q-table and updates it based on local observations of channel quality, task size, and energy consumption.
• The algorithm uses ε-greedy exploration to balance exploitation of learned actions and exploration of new strategies.
• The system models dynamic user mobility, random task arrivals, and varying channel conditions to reflect real-world IoT edge environments.
• Simulations evaluate performance across varying numbers of users, access technologies (LoRa, WiFi), and channel configurations.

Experimental results

Research questions

• RQ1 How can computation offloading be optimized in dynamic, resource-constrained IoT edge networks with multiple competing users?
• RQ2 Can a decentralized multi-agent reinforcement learning approach achieve better system cost and energy efficiency than centralized or random offloading strategies?
• RQ3 To what extent does the IL-based MA-Q algorithm converge and maintain stability under increasing user density and channel contention?
• RQ4 How does the proposed method compare to centralized approaches in terms of system cost and gateway overhead?
• RQ5 What is the impact of different exploration rates (ε) on the convergence and performance of the learning algorithm?

Key findings

• The IL-based MA-Q algorithm converges stably over time, with average cost per time slot decreasing as training episodes increase.
• The algorithm achieves lower average system cost than both the Random and Centralized benchmark algorithms when the number of end users is less than 27.
• With higher user counts (e.g., 30 users), the IL-based MA-Q algorithm incurs slightly higher cost than the Centralized approach but avoids the need for centralized channel estimation.
• The method effectively prevents radio access collisions, with 20 out of 30 users successfully offloading tasks using LoRa and WiFi channels without interference.
• The algorithm demonstrates robustness across varying exploration rates, with ε = 0.9 leading to higher cost due to excessive exploration.
• The distributed nature of the algorithm significantly reduces computational and signaling load on the gateway compared to centralized methods.