[Paper Review] Integrating LEO Satellites and Multi-UAV Reinforcement Learning for Hybrid FSO/RF Non-Terrestrial Networks

Ju-Hyung Lee, Jihong Park+2 more|arXiv (Cornell University)|Oct 20, 2020

UAV Applications and Optimization45 references4 citations

TL;DR

This paper proposes a hybrid FSO/RF non-terrestrial network integrating LEO satellites and multi-UAV relays using multi-agent deep reinforcement learning (MARL) to optimize real-time satellite-UAV associations, UAV trajectories, and link selection. The scheme achieves 1.99× higher end-to-end sum throughput and 2.25× better energy efficiency than benchmarks, with hybrid FSO/RF links boosting peak and worst-case throughput by up to 62.56× and 21.09×, respectively.

ABSTRACT

A mega-constellation of low-altitude earth orbit (LEO) satellites (SATs) and burgeoning unmanned aerial vehicles (UAVs) are promising enablers for high-speed and long-distance communications in beyond fifth-generation (5G) systems. Integrating SATs and UAVs within a non-terrestrial network (NTN), in this article we investigate the problem of forwarding packets between two faraway ground terminals through SAT and UAV relays using either millimeter-wave (mmWave) radio-frequency (RF) or free-space optical (FSO) link. Towards maximizing the communication efficiency, the real-time associations with orbiting SATs and the moving trajectories of UAVs should be optimized with suitable FSO/RF links, which is challenging due to the time-varying network topology and a huge number of possible control actions. To overcome the difficulty, we lift this problem to multi-agent deep reinforcement learning (MARL) with a novel action dimensionality reduction technique. Simulation results corroborate that our proposed SAT-UAV integrated scheme achieves 1.99x higher end-to-end sum throughput compared to a benchmark scheme with fixed ground relays. While improving the throughput, our proposed scheme also aims to reduce the UAV control energy, yielding 2.25x higher energy efficiency than a baseline method only maximizing the throughput. Lastly, thanks to utilizing hybrid FSO/RF links, the proposed scheme achieves up to 62.56x higher peak throughput and 21.09x higher worst-case throughput than the cases utilizing either RF or FSO links, highlighting the importance of co-designing SAT-UAV associations, UAV trajectories, and hybrid FSO/RF links in beyond-5G NTNs.

Motivation & Objective

• Address the challenge of dynamic topology in non-terrestrial networks (NTNs) with moving LEO satellites and UAVs.
• Maximize end-to-end throughput while minimizing UAV control energy in long-distance, multi-hop satellite-UAV-ground relay systems.
• Co-design UAV trajectory control, satellite-UAV associations, and hybrid FSO/RF link selection to enhance communication efficiency.
• Overcome the scalability and computational complexity of real-time control in high-dimensional, time-varying NTN environments.
• Demonstrate the superiority of hybrid FSO/RF over single-mode RF or FSO links in terms of peak and worst-case throughput performance.

Proposed method

• Formulates the SAT-UAV relay problem as a multi-agent deep reinforcement learning (MARL) task with a centralized critic architecture.
• Introduces a novel action dimensionality reduction technique to manage the large action space arising from dynamic UAV positions and satellite associations.
• Models the system using a Markov decision process where UAVs learn optimal trajectories and link-layer decisions (FSO or RF) based on channel conditions and link distances.
• Employs spectral efficiency models for mmWave RF (Equation 6) and FSO (Equation 10) to guide link selection based on real-time distance and signal-to-noise ratio.
• Uses a reward function that jointly optimizes end-to-end sum throughput and energy efficiency (EE), balancing data rate and UAV control cost.
• Trains the MARL agent via deep Q-networks (DQN) with experience replay and target networks, enabling stable learning in dynamic, non-stationary environments.

Experimental results

Research questions

• RQ1 How can real-time associations between LEO satellites and UAVs be optimized to maximize end-to-end throughput in a dynamic NTN?
• RQ2 What is the impact of UAV trajectory control on system throughput and energy efficiency in hybrid FSO/RF satellite-UAV relay networks?
• RQ3 How does the integration of hybrid FSO/RF links improve performance compared to using only RF or only FSO links in long-distance NTN scenarios?
• RQ4 To what extent does the proposed MARL framework with action dimensionality reduction outperform fixed-relay or single-mode link baselines?
• RQ5 How do varying link distances and channel conditions influence the optimal selection between FSO and RF links in the SAT-UAV chain?

Key findings

• The proposed SAT-UAV integrated scheme achieves 1.99× higher end-to-end sum throughput compared to a benchmark with fixed ground relays.
• The scheme improves energy efficiency by 2.25× compared to a baseline that only maximizes throughput, demonstrating effective UAV control energy reduction.
• Hybrid FSO/RF links yield up to 62.56× higher peak throughput and 21.09× higher worst-case throughput than using only FSO or only RF links.
• In the SAT-UAV path, FSO is selected in 69.9% of the Src-LEO1 and LEO2-Dst links (average distance ~2099 km), while RF dominates in shorter links (e.g., 88.1% RF in LEO1-UAV2 at ~907 km).
• The FSO/RF link selection is highly dynamic and distance-dependent, with FSO outperforming RF up to ~2400 km and RF dominating beyond that distance.
• The MARL framework successfully learns to balance throughput and energy efficiency, with trajectory and association decisions highly dependent on the chosen objective function.