[Paper Review] Cooperative Internet of UAVs: Distributed Trajectory Design by Multi-agent Deep Reinforcement Learning

Jingzhi Hu, Hongliang Zhang+3 more|arXiv (Cornell University)|Jul 28, 2020

UAV Applications and Optimization37 references4 citations

TL;DR

This paper proposes a distributed trajectory design framework for cooperative unmanned aerial vehicles (UAVs) in a cellular Internet of UAVs using multi-agent deep reinforcement learning. By introducing a compound-action actor-critic (CA2C) algorithm that jointly optimizes discrete task selection and continuous location choices, the method achieves significant age of information (AoI) reduction—outperforming four baselines and enabling cooperative UAVs to reduce AoI more effectively than non-cooperative counterparts.

ABSTRACT

Due to the advantages of flexible deployment and extensive coverage, unmanned aerial vehicles (UAVs) have great potential for sensing applications in the next generation of cellular networks, which will give rise to a cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the UAVs execute sensing tasks through cooperative sensing and transmission to minimize the age of information (AoI). However, the cooperative sensing and transmission is tightly coupled with the UAVs' trajectories, which makes the trajectory design challenging. To tackle this challenge, we propose a distributed sense-and-send protocol, where the UAVs determine the trajectories by selecting from a discrete set of tasks and a continuous set of locations for sensing and transmission. Based on this protocol, we formulate the trajectory design problem for AoI minimization and propose a compound-action actor-critic (CA2C) algorithm to solve it based on deep reinforcement learning. The CA2C algorithm can learn the optimal policies for actions involving both continuous and discrete variables and is suited for the trajectory design. {Our simulation results show that the CA2C algorithm outperforms four baseline algorithms}. Also, we show that by dividing the tasks, cooperative UAVs can achieve a lower AoI compared to non-cooperative UAVs.

Motivation & Objective

• To address the challenge of minimizing age of information (AoI) in a cellular Internet of UAVs where multiple UAVs perform cooperative sensing and transmission.
• To design a distributed trajectory planning protocol that enables UAVs to jointly optimize sensing locations and transmission schedules without centralized coordination.
• To develop a reinforcement learning-based solution capable of handling mixed discrete-continuous action spaces for efficient trajectory policy learning.
• To evaluate the performance gain of cooperative UAVs over non-cooperative schemes in terms of AoI minimization and system efficiency.
• To investigate the impact of flight altitude and subcarrier allocation on AoI performance in a 3D UAV trajectory design setting.

Proposed method

• Proposes a distributed sense-and-send protocol where UAVs select from a discrete set of tasks and continuous spatial locations for sensing and transmission.
• Models the trajectory design problem as a Markov Decision Process (MDP) with the goal of minimizing normalized accumulated AoI across all tasks.
• Introduces a compound-action actor-critic (CA2C) algorithm combining Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) to handle joint discrete-continuous action spaces.
• Uses a centralized critic with decentralized actors to enable multi-agent coordination while maintaining scalability and distributed learning.
• Employs experience replay and target networks to stabilize training, with action outputs including both task indices and 3D location coordinates.
• Extends the framework to 3D trajectory design by allowing UAVs to choose sensing altitudes within a defined range [100, 200] m.

Experimental results

Research questions

• RQ1 How can cooperative UAVs be coordinated to minimize the accumulated age of information (AoI) in a cellular Internet of UAVs?
• RQ2 What is the performance gain of cooperative sensing and transmission over non-cooperative UAV deployment in terms of AoI reduction?
• RQ3 How does the optimal flight altitude for AoI minimization vary with the number of available subcarriers?
• RQ4 Can a deep reinforcement learning algorithm effectively handle mixed discrete-continuous action spaces in multi-agent UAV trajectory design?
• RQ5 How does the proposed CA2C algorithm compare to conventional trajectory planning methods like shortest route and greedy algorithms in AoI performance?

Key findings

• The proposed CA2C algorithm outperforms four baseline algorithms—greedy, shortest route, non-cooperative multi-agent DRL, and centralized DRL—in minimizing normalized accumulated AoI.
• Cooperative UAVs achieve a lower normalized accumulated AoI than non-cooperative UAVs, with the AoI reduction increasing as the number of UAVs increases.
• The optimal flight altitude for minimizing AoI is 150 m when 80 subcarriers are available, and this optimal altitude increases with the number of subcarriers.
• With K=80 subcarriers, the average sensing altitude selected by UAVs using the CA2C algorithm is 161 m, close to the optimal 150 m identified in simulations.
• The normalized accumulated AoI decreases as the number of subcarriers increases, due to shorter transmission durations.
• Conventional algorithms like shortest route and greedy result in overlapping UAV trajectories and higher AoI due to inefficient use of environmental state information.