[Paper Review] Cellular-Connected UAVs over 5G: Deep Reinforcement Learning for Interference Management

Ursula Challita, Walid Saad+1 more|arXiv (Cornell University)|Jan 16, 2018

UAV Applications and Optimization25 references97 citations

TL;DR

The paper proposes a deep reinforcement learning framework based on echo state networks to enable interference-aware path planning and resource management for multiple cellular-connected UAVs, achieving SPNE in a dynamic game.

ABSTRACT

In this paper, an interference-aware path planning scheme for a network of cellular-connected unmanned aerial vehicles (UAVs) is proposed. In particular, each UAV aims at achieving a tradeoff between maximizing energy efficiency and minimizing both wireless latency and the interference level caused on the ground network along its path. The problem is cast as a dynamic game among UAVs. To solve this game, a deep reinforcement learning algorithm, based on echo state network (ESN) cells, is proposed. The introduced deep ESN architecture is trained to allow each UAV to map each observation of the network state to an action, with the goal of minimizing a sequence of time-dependent utility functions. Each UAV uses ESN to learn its optimal path, transmission power level, and cell association vector at different locations along its path. The proposed algorithm is shown to reach a subgame perfect Nash equilibrium (SPNE) upon convergence. Moreover, an upper and lower bound for the altitude of the UAVs is derived thus reducing the computational complexity of the proposed algorithm. Simulation results show that the proposed scheme achieves better wireless latency per UAV and rate per ground user (UE) while requiring a number of steps that is comparable to a heuristic baseline that considers moving via the shortest distance towards the corresponding destinations. The results also show that the optimal altitude of the UAVs varies based on the ground network density and the UE data rate requirements and plays a vital role in minimizing the interference level on the ground UEs as well as the wireless transmission delay of the UAV.

Motivation & Objective

• Address interference management in cellular-connected UAV networks.
• Develop online, adaptive path planning that balances energy efficiency, latency, and ground-cell interference.
• Formulate the problem as a dynamic noncooperative game among UAVs.
• Enable autonomous UAVs to learn optimal paths, power, and BS association along their trajectories.
• Derive altitude bounds to reduce computational complexity while ensuring SINR and latency targets.

Proposed method

• Model the UAV network as a dynamic noncooperative game with UAVs as players.
• Introduce a deep reinforcement learning approach using echo state network (ESN) cells for SPNE learning.
• Define observations, actions, and rewards that couple UAV trajectory, power, and BS association.
• Use a multi-layer deep ESN to capture temporal dependencies and learn policies.
• Derive analytical bounds on UAV altitudes to constrain the action space and improve efficiency.
• Provide simulation results showing tradeoffs among energy efficiency, latency, and ground interference.

Experimental results

Research questions

• RQ1 How can cellular-connected UAVs autonomously plan trajectories that minimize interference to ground UEs while meeting latency requirements?
• RQ2 Can a deep ESN-based RL framework converge to a subgame perfect Nash equilibrium in a multi-UAV interference-limited setting?
• RQ3 How do altitude and 3D positioning affect network performance and interference with ground cells?
• RQ4 What are the impacts of ground network density and UE data rate requirements on optimal UAV altitude and trajectory?

Key findings

• The proposed ESN-based RL framework achieves a subgame perfect Nash equilibrium upon convergence.
• Simulation results show improvements in wireless latency per UAV and rate per ground user compared to a shortest-path baseline.
• Upper and lower altitude bounds are derived, reducing the algorithm’s computational complexity.
• UAV altitude is shown to depend on ground network density and UE data rate requirements and is critical for minimizing interference and delay.