[Paper Review] Explainable and Safe Reinforcement Learning for Autonomous Air Mobility

Lei Wang, Yang Hongyu+3 more|arXiv (Cornell University)|Nov 24, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes SafeDQN-X, a fully explainable deep reinforcement learning framework for autonomous air traffic control that decouples safety and efficiency objectives using separate Q-value networks. By training a safety-aware DQN and a goal-oriented DQN independently and combining their outputs, the method improves conflict resolution performance, enhances transparency through interpretable safety value functions, and reveals vulnerabilities to targeted adversarial attacks that outperform uniform attacks with only four attacks instead of full-time interference.

ABSTRACT

Increasing traffic demands, higher levels of automation, and communication enhancements provide novel design opportunities for future air traffic controllers (ATCs). This article presents a novel deep reinforcement learning (DRL) controller to aid conflict resolution for autonomous free flight. Although DRL has achieved important advancements in this field, the existing works pay little attention to the explainability and safety issues related to DRL controllers, particularly the safety under adversarial attacks. To address those two issues, we design a fully explainable DRL framework wherein we: 1) decompose the coupled Q value learning model into a safety-awareness and efficiency (reach the target) one; and 2) use information from surrounding intruders as inputs, eliminating the needs of central controllers. In our simulated experiments, we show that by decoupling the safety-awareness and efficiency, we can exceed performance on free flight control tasks while dramatically improving explainability on practical. In addition, the safety Q learning module provides rich information about the safety situation of environments. To study the safety under adversarial attacks, we additionally propose an adversarial attack strategy that can impose both safety-oriented and efficiency-oriented attacks. The adversarial aims to minimize safety/efficiency by only attacking the agent at a few time steps. In the experiments, our attack strategy increases as many collisions as the uniform attack (i.e., attacking at every time step) by only attacking the agent four times less often, which provide insights into the capabilities and restrictions of the DRL in future ATC designs. The source code is publicly available at https://github.com/WLeiiiii/Gym-ATC-Attack-Project.

Motivation & Objective

• To address the lack of explainability and safety robustness in existing deep reinforcement learning (DRL) controllers for autonomous air traffic control (ATC).
• To develop a DRL framework that enables transparent decision-making by decomposing safety and efficiency objectives into separate value functions.
• To evaluate the safety of the DRL agent under adversarial attacks, particularly those targeting only a few critical time steps.
• To demonstrate that local, neighbor-based information suffices for effective conflict resolution without centralized coordination.
• To reveal structural vulnerabilities in explainable DRL designs through a targeted adversarial attack strategy.

Proposed method

• Decompose the traditional DQN into two independent networks: a Safe DQN for assessing environmental safety and a Goal DQN for maximizing target reach efficiency.
• Use dueling DQN architecture to estimate state values and advantage functions separately, enabling interpretable safety and goal value estimation.
• Train the Safe DQN using a safety-oriented reward function that penalizes proximity to other aircraft and rewards safe separation.
• Combine actions from both networks using a weighted fusion strategy, where safety takes precedence in high-risk states.
• Design a targeted adversarial attack strategy that injects perturbations only at critical decision points to maximize impact on safety or efficiency.
• Visualize safety and action probability maps using Q-value heatmaps to demonstrate model interpretability and decision transparency.

Experimental results

Research questions

• RQ1 Can decoupling safety and efficiency in DRL improve both performance and explainability in autonomous conflict resolution?
• RQ2 How can safety be quantitatively measured and visualized in real-time air traffic control using deep reinforcement learning?
• RQ3 To what extent is a DRL-based ATC controller vulnerable to adversarial attacks that target only a few time steps?
• RQ4 Can local, neighbor-based information replace global surveillance data for effective conflict resolution in free flight?
• RQ5 Does a transparent, modular DRL design expose structural weaknesses that can be exploited by intelligent adversaries?

Key findings

• SafeDQN-X outperforms standard DQN in conflict resolution tasks by decoupling safety and efficiency, achieving higher safety and improved convergence.
• The safety value function learned by Safe DQN provides a reliable, interpretable metric of environmental safety, with values of 0.1352 in safe states and -0.0013 in dangerous states.
• The proposed adversarial attack strategy achieves as many collisions as a uniform attack (attacking every time step) while only attacking the agent four times less frequently.
• In safe states, all actions have similar safety probabilities, and the goal DQN dominates decision-making; in dangerous states, the safety DQN overrides the goal DQN to prioritize avoidance.
• The final action is a balanced choice—acceleration with stable heading—when safety and goal preferences conflict, demonstrating effective fusion of dual objectives.
• The study reveals that structural transparency in DRL models can inadvertently expose design vulnerabilities, making them susceptible to highly targeted adversarial attacks.