[Paper Review] Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networks

Jingjing Cui, Yuanwei Liu+1 more|arXiv (Cornell University)|Oct 24, 2018

UAV Applications and OptimizationEngineering34 references17 citations

TL;DR

This paper proposes a multi-agent reinforcement learning (MARL) framework for dynamic resource allocation in UAV networks, where each UAV independently selects users, power levels, and subchannels using a decentralized Q-learning approach. The method achieves near-optimal performance with minimal inter-UAV communication, balancing performance gains and overhead in stochastic, uncertain environments.

ABSTRACT

Unmanned aerial vehicles (UAVs) are capable of serving as aerial base stations (BSs) for providing both cost-effective and on-demand wireless communications. This article investigates dynamic resource allocation of multiple UAVs enabled communication networks with the goal of maximizing long-term rewards. More particularly, each UAV communicates with a ground user by automatically selecting its communicating users, power levels and subchannels without any information exchange among UAVs. To model the uncertainty of environments, we formulate the long-term resource allocation problem as a stochastic game for maximizing the expected rewards, where each UAV becomes a learning agent and each resource allocation solution corresponds to an action taken by the UAVs. Afterwards, we develop a multi-agent reinforcement learning (MARL) framework that each agent discovers its best strategy according to its local observations using learning. More specifically, we propose an agent-independent method, for which all agents conduct a decision algorithm independently but share a common structure based on Q-learning. Finally, simulation results reveal that: 1) appropriate parameters for exploitation and exploration are capable of enhancing the performance of the proposed MARL based resource allocation algorithm; 2) the proposed MARL algorithm provides acceptable performance compared to the case with complete information exchanges among UAVs. By doing so, it strikes a good tradeoff between performance gains and information exchange overheads.

Motivation & Objective

• To address dynamic resource allocation in multi-UAV networks with minimal inter-UAV information exchange.
• To model the resource allocation problem as a stochastic game to maximize long-term rewards.
• To develop a decentralized MARL algorithm enabling each UAV to learn optimal strategies based on local observations.
• To evaluate the trade-off between performance and communication overhead in dynamic UAV networks.

Proposed method

• Formulates the resource allocation problem as a stochastic game where each UAV acts as an independent learning agent.
• Employs an agent-independent MARL method based on Q-learning, with shared architecture but independent decision-making.
• Uses a Q-function update rule that incorporates state transitions, rewards, and a discount factor δ to model long-term value.
• Applies contraction mapping theory to prove convergence of the Q-value function to the optimal solution.
• Introduces a learning update rule that ensures bounded variance and almost-sure convergence to the optimal policy.
• Employs a stochastic approximation framework to handle environmental uncertainty and non-stationarity in UAV networks.

Experimental results

Research questions

• RQ1 Can a decentralized MARL approach achieve near-optimal resource allocation in multi-UAV networks without full information exchange?
• RQ2 How do exploration and exploitation trade-offs affect the performance of the MARL-based resource allocation algorithm?
• RQ3 What is the performance gap between the proposed decentralized MARL and a centralized scheme with complete information exchange?
• RQ4 Does the proposed MARL framework converge to the optimal policy under environmental uncertainty?
• RQ5 How does the algorithm balance performance gains with communication overhead in dynamic UAV networks?

Key findings

• Appropriate tuning of exploration and exploitation parameters significantly improves the performance of the proposed MARL algorithm.
• The MARL-based approach achieves performance close to that of a centralized scheme with complete information exchange.
• The algorithm strikes a favorable trade-off between performance and communication overhead, making it suitable for real-time UAV networks.
• Theoretical analysis confirms almost-sure convergence of the Q-value function to the optimal solution under the proposed learning rule.
• The variance of the learning update is bounded, ensuring stable and reliable convergence in stochastic environments.
• The contraction mapping property of the Q-update operator guarantees convergence to the optimal policy.