[Paper Review] Multi-UAV Speed Control with Collision Avoidance and Handover-aware Cell Association: DRL with Action Branching

Zijiang Yan, Wael Jaafar+2 more|arXiv (Cornell University)|Jul 24, 2023

Autonomous Vehicle Technology and SafetyEngineering3 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL) framework with action branching to jointly optimize multi-UAV speed control, cell association, and collision avoidance on a 3D aerial highway. Using a Branching Dueling Q-Network (BDQ) and its double variant (BDDQN), the method achieves 18.32% improvement over benchmarks in balancing traffic flow and HO-aware communication performance.

ABSTRACT

This paper presents a deep reinforcement learning solution for optimizing multi-UAV cell-association decisions and their moving velocity on a 3D aerial highway. The objective is to enhance transportation and communication performance, including collision avoidance, connectivity, and handovers. The problem is formulated as a Markov decision process (MDP) with UAVs' states defined by velocities and communication data rates. We propose a neural architecture with a shared decision module and multiple network branches, each dedicated to a specific action dimension in a 2D transportation-communication space. This design efficiently handles the multi-dimensional action space, allowing independence for individual action dimensions. We introduce two models, Branching Dueling Q-Network (BDQ) and Branching Dueling Double Deep Q-Network (Dueling DDQN), to demonstrate the approach. Simulation results show a significant improvement of 18.32% compared to existing benchmarks.

Motivation & Objective

• Address the lack of integrated optimization for multi-UAV motion dynamics and communication performance in 3D aerial highways.
• Simultaneously maximize aerial traffic flow and minimize handover (HO) losses while ensuring collision avoidance.
• Develop a DRL-based solution that jointly handles UAV speed, lane changes, and base station (BS) assignments under dynamic conditions.
• Overcome limitations of existing Q-learning-based methods in convergence and policy quality for multi-dimensional action spaces.
• Enable robust, scalable decision-making for UAVs in high-density aerial traffic with dynamic wireless connectivity requirements.

Proposed method

• Formulate the multi-UAV control problem as a Markov Decision Process (MDP) with states defined by UAV speed and communication data rates.
• Design a 2D action space encompassing UAV acceleration/deceleration, lane changes, and BS assignment decisions.
• Introduce a neural architecture with a shared decision module and multiple independent branches—one per action dimension—to handle multi-dimensional actions efficiently.
• Propose two DRL agents: Branching Dueling Q-Network (BDQ) and Branching Dueling Double DQN (BDDQN), enhancing exploration-exploitation and stability.
• Use a fully connected feed-forward network (FNN) with ReLU activations to approximate Q-values, followed by branching heads for individual action outputs.
• Apply linear activation at the output layer and train using experience replay and target networks to improve learning stability.

Experimental results

Research questions

• RQ1 How can multi-UAV speed control, lane changes, and cell association be jointly optimized to maximize both traffic flow and communication quality?
• RQ2 What is the impact of UAV speed on collision frequency, handover rates, and data rate performance in a 3D aerial highway environment?
• RQ3 Can an action branching DRL architecture outperform conventional DQN and DDQN in handling multi-dimensional UAV control actions?
• RQ4 How does the number of available base stations affect the trade-off between communication reward and handover frequency?
• RQ5 What is the optimal balance between maximizing data rates and minimizing handover-related penalties in dynamic UAV networks?

Key findings

• The proposed BDDQN algorithm achieves an 18.32% improvement in overall performance compared to existing benchmarks, including DDQN and SDB.
• At a target speed of 10 m/s, the communication reward is maximized, while higher speeds (e.g., 20 m/s) lead to increased collisions and reduced transportation rewards.
• BDDQN converges to a handover rate below 1% after ~1000 training episodes, indicating effective learning of HO avoidance.
• With 15 BSs, the communication reward peaks for BDQN and BDDQN, but further increasing BS density leads to higher HO penalties due to excessive cell reselection.
• When only 5 BSs are available, the HO rate is higher than with 15 BSs, as the algorithm prioritizes reaching target speed and thus triggers more handovers.
• BDDQN outperforms the SDB benchmark by 16.7% in transportation reward, 23.4% in communication reward, and 10.9% in HO rate at 10 m/s speed.