[Paper Review] A 3D Game Theoretical Framework for the Evaluation of Unmanned Aircraft Systems Airspace Integration Concepts

Negin Musavi, Ayman Manzoor+1 more|arXiv (Cornell University)|Feb 20, 2018

Aerospace and Aviation Technology40 references3 citations

TL;DR

This paper proposes a 3D game-theoretical simulation framework that models dynamic human pilot decision-making using dynamic level-k reasoning and Neural Fitted Q-Learning to evaluate UAS integration into the NAS. It demonstrates that shared conflict resolution responsibility between manned aircraft and UAS minimizes separation violations and optimizes performance, outperforming scenarios where only one platform is responsible.

ABSTRACT

Predicting the outcomes of integrating Unmanned Aerial Systems (UAS) into the National Airspace System (NAS) is a complex problem which is required to be addressed by simulation studies before allowing the routine access of UAS into the NAS. This paper focuses on providing a 3-dimensional (3D) simulation framework using a game theoretical methodology to evaluate integration concepts using scenarios where manned and unmanned air vehicles co-exist. In the proposed method, human pilot interactive decision making process is incorporated into airspace models which can fill the gap in the literature where the pilot behavior is generally assumed to be known a priori. The proposed human pilot behavior is modeled using dynamic level-k reasoning concept and approximate reinforcement learning. The level-k reasoning concept is a notion in game theory and is based on the assumption that humans have various levels of decision making. In the conventional "static" approach, each agent makes assumptions about his or her opponents and chooses his or her actions accordingly. On the other hand, in the dynamic level-k reasoning, agents can update their beliefs about their opponents and revise their level-k rule. In this study, Neural Fitted Q Iteration, which is an approximate reinforcement learning method, is used to model time-extended decisions of pilots with 3D maneuvers. An analysis of UAS integration is conducted using an example 3D scenario in the presence of manned aircraft and fully autonomous UAS equipped with sense and avoid algorithms.

Motivation & Objective

• To address the lack of realistic human pilot behavior modeling in UAS airspace integration simulations, where pilots are often assumed to act deterministically.
• To overcome limitations of prior 2D models and static policy assumptions in hybrid airspace simulation frameworks.
• To develop a 3D simulation environment that captures time-extended, adaptive decision-making by pilots with varying cognitive levels.
• To evaluate the impact of conflict resolution responsibility assignment (manned vs. UAS) on safety and performance metrics in UAS-NAS integration.
• To enable quantitative assessment of UAS sense-and-avoid algorithms, separation standards, and system parameters using a behavior-aware simulation framework.

Proposed method

• Employs dynamic level-k reasoning to model heterogeneous pilot cognitive levels (level-0 to level-2), allowing agents to update beliefs and revise strategies during repeated interactions.
• Integrates Neural Fitted Q-Learning (NFQ) to model time-extended, sequential decision-making in 3D airspace, replacing static Q-tables with deep function approximation.
• Simulates hybrid airspace scenarios with manned aircraft (using dynamic level-k policies) and fully autonomous UAS equipped with SAA1 and SAA2 sense-and-avoid algorithms.
• Models conflict resolution responsibility as a configurable parameter: only manned aircraft, only UAS, or shared responsibility, with agents following respective maneuvering rules.
• Uses a 3D airspace model with realistic kinematics and conflict detection based on minimum separation standards.
• Employs a safety metric (separation violations) and performance metric (trajectory deviation and flight time) to evaluate system outcomes.

Experimental results

Research questions

• RQ1 How does dynamic level-k reasoning improve the realism and predictive power of pilot behavior modeling in UAS-NAS integration simulations compared to static or deterministic models?
• RQ2 What is the impact of assigning conflict resolution responsibility to different platforms (manned aircraft, UAS, or both) on safety and performance in 3D single-encounter scenarios?
• RQ3 How do different SAA algorithms (SAA1 and SAA2) perform under varying responsibility assignment and pilot cognitive level distributions?
• RQ4 To what extent does the use of Neural Fitted Q-Learning enable scalable modeling of adaptive, time-extended decisions in 3D airspace without relying on large Q-tables?
• RQ5 How do varying pilot cognitive levels (10% level-0, 60% level-1, 30% level-2) affect overall system safety and performance in shared conflict resolution scenarios?

Key findings

• The shared conflict resolution responsibility scenario resulted in the lowest number of separation violations—minimizing safety risks—across both SAA1 and SAA2 algorithms.
• When only manned aircraft were responsible, they exhibited greater average trajectory deviation than in shared-responsibility cases, indicating increased workload and less efficient conflict resolution.
• When UAS were solely responsible, they showed higher average trajectory deviations compared to shared responsibility, suggesting inefficiency or suboptimal maneuvering under sole accountability.
• UAS flight times were shortest when only manned aircraft were responsible for conflict resolution, indicating that UAS could maintain optimal paths when not required to maneuver.
• The dynamic level-k reasoning model successfully captured adaptive pilot behavior, with level-1 and level-2 pilots updating beliefs and adjusting strategies during repeated interactions.
• The integration of Neural Fitted Q-Learning enabled scalable, function-based policy learning without storing large Q-tables, allowing modeling of complex 3D, time-extended decision sequences.