[Paper Review] UAV-to-Device Underlay Communications: Age of Information Minimization by Multi-agent Deep Reinforcement Learning

Fanyi Wu, Hongliang Zhang+4 more|arXiv (Cornell University)|Mar 12, 2020

Age of Information Optimization30 references4 citations

TL;DR

This paper proposes a multi-agent deep reinforcement learning (DRL) framework to minimize age of information (AoI) in UAV-to-Device (U2D) underlay communications within a cellular Internet of UAVs. By modeling UAV trajectory design as a Markov decision process with continuous state-action spaces, the authors employ a deep deterministic policy gradient (DDPG)-based multi-agent DRL algorithm to jointly optimize UAV trajectories for sensing and transmission, achieving lower AoI than greedy and policy gradient baselines in simulations.

ABSTRACT

In recent years, unmanned aerial vehicles (UAVs) have found numerous sensing applications, which are expected to add billions of dollars to the world economy in the next decade. To further improve the Quality-of-Service (QoS) in such applications, the 3rd Generation Partnership Project (3GPP) has considered the adoption of terrestrial cellular networks to support UAV sensing services, also known as the cellular Internet of UAVs. In this paper, we consider a cellular Internet of UAVs, where the sensory data can be transmitted either to base station (BS) via cellular links, or to mobile devices by underlay UAV-to-Device (U2D) communications. To evaluate the freshness of data, the age of information (AoI) is adopted, in which a lower AoI implies fresher data. Since UAVs' AoIs are determined by their trajectories during sensing and transmission, we investigate the AoI minimization problem for UAVs by designing their trajectories. This problem is a Markov decision problem (MDP) with an infinite state-action space, and thus we utilize multi-agent deep reinforcement learning (DRL) to approximate the state-action space. Then, we propose a multi-UAV trajectory design algorithm to solve this problem. Simulation results show that our algorithm achieves a lower AoI than greedy algorithm and policy gradient algorithm.

Motivation & Objective

• To address the challenge of minimizing age of information (AoI) in UAV-to-Device (U2D) underlay communications within a cellular Internet of UAVs.
• To jointly optimize UAV trajectories for sensing and transmission, as AoI depends on both mobility and transmission dynamics.
• To handle the infinite state-action space and non-observable interference in a multi-UAV system, which complicates traditional model-based methods.
• To design a joint sensing and transmission protocol that enables Markovian state transitions for modeling the AoI minimization problem as a Markov decision process (MDP).
• To develop a scalable, distributed trajectory design algorithm using multi-agent deep reinforcement learning (DRL) for real-time adaptation in dynamic UAV networks.

Proposed method

• Formulates the AoI minimization problem as a Markov decision process (MDP) with continuous state and action spaces due to UAV mobility and transmission power control.
• Models UAV sensing and transmission as state transitions in a Markov chain to capture temporal dependencies in AoI evolution.
• Applies multi-agent deep reinforcement learning (DRL) with a deep deterministic policy gradient (DDPG) algorithm to learn decentralized policies for each UAV.
• Uses a centralized critic network to estimate the value function while each UAV acts based on its local observation, enabling credit assignment in multi-agent settings.
• Implements a joint sensing and transmission protocol to coordinate UAV operations and ensure data freshness through timely updates.
• Employs a reward function that penalizes high AoI and encourages frequent, reliable data updates to guide policy learning.

Experimental results

Research questions

• RQ1 How can UAV trajectories be jointly optimized for sensing and transmission to minimize age of information (AoI) in a multi-UAV cellular network?
• RQ2 What is the impact of underlay U2D communications on AoI when interference from other UAVs is unobservable and dynamic?
• RQ3 Can multi-agent deep reinforcement learning effectively handle the infinite state-action space and partial observability in UAV trajectory design for AoI minimization?
• RQ4 How does the proposed DRL-based trajectory design compare to greedy and policy gradient baselines in terms of AoI performance?
• RQ5 What is the relationship between AoI and system parameters such as data demand and subchannel availability?

Key findings

• The proposed multi-agent DRL algorithm achieves a lower average AoI than both the greedy algorithm and the policy gradient baseline in all tested scenarios.
• UAV AoI increases linearly with the sensory data demand, indicating a direct trade-off between data volume and freshness.
• AoI decreases with the number of available subchannels, but eventually saturates due to interference limits and resource constraints.
• The algorithm effectively balances UAV mobility and transmission scheduling, reducing data staleness through coordinated trajectory planning.
• Simulation results confirm that the DRL-based approach adapts well to dynamic environments and outperforms heuristic methods in minimizing data age.
• The method successfully handles the continuous, high-dimensional state-action space and partial observability, demonstrating scalability and robustness in multi-UAV systems.