[论文解读] Cellular-Connected UAVs over 5G: Deep Reinforcement Learning for Interference Management

Ursula Challita, Walid Saad+1 more|arXiv (Cornell University)|Jan 16, 2018

UAV Applications and Optimization参考文献 25被引用 97

一句话总结

本论文提出基于回声状态网络的深度强化学习框架，使多台蜂窝连接的无人机具干扰感知的路径规划和资源管理，在动态博弈中实现子博弈完美纳什均衡（SPNE）。

ABSTRACT

In this paper, an interference-aware path planning scheme for a network of cellular-connected unmanned aerial vehicles (UAVs) is proposed. In particular, each UAV aims at achieving a tradeoff between maximizing energy efficiency and minimizing both wireless latency and the interference level caused on the ground network along its path. The problem is cast as a dynamic game among UAVs. To solve this game, a deep reinforcement learning algorithm, based on echo state network (ESN) cells, is proposed. The introduced deep ESN architecture is trained to allow each UAV to map each observation of the network state to an action, with the goal of minimizing a sequence of time-dependent utility functions. Each UAV uses ESN to learn its optimal path, transmission power level, and cell association vector at different locations along its path. The proposed algorithm is shown to reach a subgame perfect Nash equilibrium (SPNE) upon convergence. Moreover, an upper and lower bound for the altitude of the UAVs is derived thus reducing the computational complexity of the proposed algorithm. Simulation results show that the proposed scheme achieves better wireless latency per UAV and rate per ground user (UE) while requiring a number of steps that is comparable to a heuristic baseline that considers moving via the shortest distance towards the corresponding destinations. The results also show that the optimal altitude of the UAVs varies based on the ground network density and the UE data rate requirements and plays a vital role in minimizing the interference level on the ground UEs as well as the wireless transmission delay of the UAV.

研究动机与目标

• 处理蜂窝连接 UAV 网络中的干扰管理。
• 开发在线、适应性路径规划，在能源效率、延迟和地面小区干扰之间取得平衡。
• 将问题建模为 UAV 之间的动态非合作博弈。
• 使自治 UAV 在其轨迹上学习最优路径、功率和基站关联。
• 推导高度界限以在确保 SINR 和延迟目标的同时降低计算复杂度。

提出的方法

• 将 UAV 网络建模为以 UAV 为参与者的动态非合作博弈。
• 引入基于回声状态网络 (ESN) 单元的深度强化学习方法用于 SPNE 学习。
• 定义将 UAV 轨迹、功率和基站关联耦合在一起的观测、动作和奖励。
• 使用多层深度 ESN 捕捉时间相关性并学习策略。
• 推导 UAV 高度的解析界限以约束动作空间并提高效率。
• 给出仿真结果，显示能源效率、延迟和地面干扰之间的权衡。

实验结果

研究问题

• RQ1 蜂窝连接的 UAV 如何在满足延迟要求的同时，自治地规划轨迹以尽量减少对地面用户的干扰？
• RQ2 基于深度 ESN 的 RL 框架是否能在多 UAV 相互干扰受限的场景中收敛到子博弈完美纳什均衡？
• RQ3 高度和三维定位如何影响网络性能及对地面小区的干扰？
• RQ4 地面网络密度和 UE 数据率需求对最优 UAV 高度与轨迹有何影响？

主要发现

• 提出的 ESN 基 RL 框架在收敛后实现了子博弈完美纳什均衡。
• 仿真结果显示与最短路径基线相比，单个 UAV 的无线时延和地面用户的速率有所提升。
• 推导出高度的上界和下界，降低算法的计算复杂度。
• UAV 高度显示出取决于地面网络密度和 UE 数据率需求，并且对最小化干扰和延迟至关重要。