[论文解读] Trajectory Design and Power Control for Multi-UAV Assisted Wireless Networks: A Machine Learning Approach

Xiao Liu, Yuanwei Liu+2 more|arXiv (Cornell University)|Dec 18, 2018

UAV Applications and Optimization参考文献 41被引用 12

一句话总结

本文提出了一种用于多无人机辅助无线网络中联合轨迹设计与功率控制的机器学习框架，采用多智能体Q-learning与回声状态网络（ESN）预测用户移动性并优化无人机定位。该方法在温和条件下实现收敛，相比基线方法获得约17%的吞吐量增益。

ABSTRACT

A novel framework is proposed for the trajectory design of multiple unmanned aerial vehicles (UAVs) based on the prediction of users' mobility information. The problem of joint trajectory design and power control is formulated for maximizing the instantaneous sum transmit rate while satisfying the rate requirement of users. In an effort to solve this pertinent problem, a three-step approach is proposed which is based on machine learning techniques to obtain both the position information of users and the trajectory design of UAVs. Firstly, a multi-agent Q-learning based placement algorithm is proposed for determining the optimal positions of the UAVs based on the initial location of the users. Secondly, in an effort to determine the mobility information of users based on a real dataset, their position data is collected from Twitter to describe the anonymous user-trajectories in the physical world. In the meantime, an echo state network (ESN) based prediction algorithm is proposed for predicting the future positions of users based on the real dataset. Thirdly, a multi-agent Q-learning based algorithm is conceived for predicting the position of UAVs in each time slot based on the movement of users. In this algorithm, multiple UAVs act as agents to find optimal actions by interacting with their environment and learn from their mistakes. Additionally, we also prove that the proposed multi-agent Q-learning based trajectory design and power control algorithm can converge under mild conditions. Numerical results are provided to demonstrate that as the size of the reservoir increases, the proposed ESN approach improves the prediction accuracy. Finally, we demonstrate that throughput gains of about 17% are achieved.

研究动机与目标

• 为解决在用户移动性约束下最大化多无人机无线网络瞬时总传输速率的挑战。
• 联合优化无人机轨迹与功率控制，以在动态环境中满足用户速率需求。
• 利用来自Twitter的真实用户移动性数据，提升用户位置预测的准确性。
• 设计一种可扩展、收敛的多智能体强化学习框架，用于无人机部署与移动性管理。

提出的方法

• 使用多智能体Q-learning算法根据初始用户位置确定最优无人机位置，将每个无人机视为独立智能体，通过环境反馈进行学习。
• 采用回声状态网络（ESN）利用基于Twitter的真实移动性数据集预测未来用户位置，提升轨迹规划精度。
• 系统将无人机建模为智能体，通过与环境的交互迭代学习最优动作（位置），利用基于吞吐量和速率约束的奖励信号。
• 采用三步框架集成用户移动性预测、无人机部署与动态轨迹优化，利用强化学习实现。
• 在温和条件下，理论上证明了多智能体Q-learning算法的收敛性，确保学习动态稳定。
• 联合优化功率控制与轨迹设计，以在满足个体用户速率需求的前提下最大化系统总速率。

实验结果

研究问题

• RQ1 在多无人机网络中，如何根据实时用户移动性模式动态优化无人机轨迹？
• RQ2 基于ESN的用户轨迹预测在多大程度上能提升无人机辅助无线网络的性能？
• RQ3 多智能体Q-learning框架能否有效协调多架无人机，在满足用户速率约束的同时最大化系统吞吐量？
• RQ4 所提出的多智能体强化学习算法在真实网络条件下表现出怎样的收敛行为？
• RQ5 通过将移动性预测与联合轨迹及功率控制相结合，可在无人机网络中实现多大的吞吐量增益？

主要发现

• 所提出的基于ESN的用户移动性预测方法在水库规模增大时表现出更高的预测精度，显示出良好的可扩展性与适应性。
• 用于无人机轨迹与功率控制的多智能体Q-learning算法在温和条件下被证明收敛，确保了稳定的学习性能。
• 与基线方案相比，吞吐量增益约为17%，验证了联合优化框架的有效性。
• 通过整合真实世界的Twitter移动性数据，能够准确建模用户移动模式，从而在真实场景中提升系统性能。
• 该框架有效平衡了用户速率需求与系统总速率最大化，在动态环境中表现出强健性。
• 强化学习的使用使得无人机能够实现自主、自适应部署，而无需预先掌握用户移动模式。