[Paper Review] A Deep Multi-Agent Reinforcement Learning Approach to Autonomous Separation Assurance

Marc Brittain, Xuxi Yang+1 more|arXiv (Cornell University)|Mar 17, 2020

Software Reliability and Analysis Research40 references26 citations

TL;DR

This paper proposes a deep multi-agent reinforcement learning framework, D2MAV-A, that enables autonomous separation assurance for air traffic in high-density, dynamic sectors using attention-augmented Proximal Policy Optimization. The framework achieves faster training, reduced speed changes, and improved scalability by learning a shared policy across agents, significantly outperforming prior methods in complex, variable-traffic scenarios.

ABSTRACT

A novel deep multi-agent reinforcement learning framework is proposed to identify and resolve conflicts among a variable number of aircraft in a high-density, stochastic, and dynamic sector. Currently the sector capacity is constrained by human air traffic controller's cognitive limitation. We investigate the feasibility of a new concept (autonomous separation assurance) and a new approach to push the sector capacity above human cognitive limitation. We propose the concept of using distributed vehicle autonomy to ensure separation, instead of a centralized sector air traffic controller. Our proposed framework utilizes Proximal Policy Optimization (PPO) that we modify to incorporate an attention network. This allows the agents to have access to variable aircraft information in the sector in a scalable, efficient approach to achieve high traffic throughput under uncertainty. Agents are trained using a centralized learning, decentralized execution scheme where one neural network is learned and shared by all agents. The proposed framework is validated on three challenging case studies in the BlueSky air traffic control environment. Numerical results show the proposed framework significantly reduces offline training time, increases performance, and results in a more efficient policy.

Motivation & Objective

• To address the limitations of human air traffic controllers in high-density airspace by enabling autonomous separation assurance using onboard AI.
• To design a scalable, real-time decision-making system that handles variable numbers of aircraft and dynamic traffic conditions.
• To improve efficiency and safety in en route and terminal airspace by minimizing speed adjustments while maintaining separation.
• To validate the framework in complex, stochastic scenarios using the BlueSky air traffic simulation environment.
• To explore transfer learning for faster convergence across diverse traffic configurations.

Proposed method

• The framework employs a centralized learning, decentralized execution scheme with a shared neural network policy across all aircraft agents.
• It integrates an attention mechanism to encode variable-length traffic information into a fixed-length context vector, enabling scalable processing of dynamic traffic.
• Proximal Policy Optimization (PPO) is used with a novel, carefully designed reward function that penalizes conflicts and rewards minimal speed changes.
• The system is trained in the BlueSky air traffic simulation environment, extended with reinforcement learning support for parallelized training.
• Transfer learning is applied by initializing the policy on a simpler case study (C) before training on a combined, more complex scenario (D).
• The framework uses parallelized training with multiple environments to accelerate policy learning and improve sample efficiency.

Experimental results

Research questions

• RQ1 Can a deep multi-agent reinforcement learning framework with attention mechanisms effectively manage autonomous separation in high-density, variable-traffic air traffic scenarios?
• RQ2 How does the inclusion of an attention network improve scalability and performance compared to non-attention baselines?
• RQ3 To what extent does transfer learning reduce training time and improve convergence in complex, multi-configuration air traffic environments?
• RQ4 Can the proposed framework significantly reduce the number of speed adjustments while maintaining conflict-free separation?
• RQ5 How does the shared policy architecture perform across varying numbers of aircraft and sector configurations?

Key findings

• The D2MAV-A framework reduced offline training time and achieved faster convergence compared to the prior D2MAV framework, particularly in complex scenarios.
• The framework reduced the number of speed change actions by 30% compared to the D2MAV baseline, indicating a more efficient policy with fewer control interventions.
• Transfer learning reduced the number of episodes to convergence on case study D from 37,172 (training from scratch) to 908, a 97.6% reduction in training steps.
• The policy trained with transfer learning achieved high performance from the start, with only a minor initial drop in performance due to adaptation to new environments.
• The attention mechanism enabled effective handling of variable numbers of aircraft and intersections without increasing model complexity.
• The framework demonstrated robustness and generalization across diverse traffic configurations, including combined scenarios with multiple case studies.