[Paper Review] Harfang3D Dog-Fight Sandbox: A Reinforcement Learning Research Platform for the Customized Control Tasks of Fighter Aircrafts

Muhammed Murat Özbek, Süleyman Yıldırım+3 more|arXiv (Cornell University)|Oct 13, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper introduces Harfang3D Dog-Fight Sandbox, a customizable, semi-realistic 3D flight simulation platform designed for reinforcement learning (RL) research in fighter aircraft control. It enables training of deep RL agents using continuous state and action spaces, with successful navigation and within-visual-range (WVR) combat tasks achieved via TD3+HER, demonstrating human-like behavior in complex aerial environments.

ABSTRACT

The advent of deep learning (DL) gave rise to significant breakthroughs in Reinforcement Learning (RL) research. Deep Reinforcement Learning (DRL) algorithms have reached super-human level skills when applied to vision-based control problems as such in Atari 2600 games where environment states were extracted from pixel information. Unfortunately, these environments are far from being applicable to highly dynamic and complex real-world tasks as in autonomous control of a fighter aircraft since these environments only involve 2D representation of a visual world. Here, we present a semi-realistic flight simulation environment Harfang3D Dog-Fight Sandbox for fighter aircrafts. It is aimed to be a flexible toolbox for the investigation of main challenges in aviation studies using Reinforcement Learning. The program provides easy access to flight dynamics model, environment states, and aerodynamics of the plane enabling user to customize any specific task in order to build intelligent decision making (control) systems via RL. The software also allows deployment of bot aircrafts and development of multi-agent tasks. This way, multiple groups of aircrafts can be configured to be competitive or cooperative agents to perform complicated tasks including Dog Fight. During the experiments, we carried out training for two different scenarios: navigating to a designated location and within visual range (WVR) combat, shortly Dog Fight. Using Deep Reinforcement Learning techniques for both scenarios, we were able to train competent agents that exhibit human-like behaviours. Based on this results, it is confirmed that Harfang3D Dog-Fight Sandbox can be utilized as a 3D realistic RL research platform.

Motivation & Objective

• To address the lack of realistic, customizable 3D simulation platforms for training RL agents in complex, dynamic aerial combat scenarios.
• To enable research on low-level control of fighter aircraft using continuous action and state spaces, without reliance on expert demonstrations or real-world data.
• To support multi-agent scenarios, including cooperative and competitive configurations, for advanced task learning such as dogfighting.
• To provide a flexible, high-fidelity simulation environment compatible with OpenAI Gym and distributed training for scalable RL experimentation.
• To validate the platform’s effectiveness through training competent RL agents on navigation and within-visual-range (WVR) combat tasks.

Proposed method

• The platform is built in C++ with Python, Lua, and Golang APIs, enabling integration with existing RL frameworks like OpenAI Gym.
• It supports multiple runtime modes, including renderless mode for efficient training without graphical overhead.
• The environment provides continuous sensory inputs including position, Euler angles, velocity, acceleration, and heading, enabling rich state representation.
• Control is applied via low-level manipulation of aileron, elevator, and rudder, allowing the agent to learn direct flight control.
• The Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Hindsight Experience Replay (HER) is used for training.
• A sparse reward function is defined based on distance to goal, with +100 for success and -100 for failure or crash, promoting efficient learning.

Experimental results

Research questions

• RQ1 Can a semi-realistic 3D flight simulation environment support effective training of deep reinforcement learning agents for complex fighter aircraft control tasks?
• RQ2 To what extent can RL agents trained in Harfang3D Dog-Fight Sandbox learn human-like navigation and dogfighting behaviors without expert demonstrations?
• RQ3 How effective is the TD3+HER algorithm in learning control policies for high-dimensional, continuous control tasks in a 3D aerial combat environment?
• RQ4 Can the platform support scalable, distributed training and multi-agent configurations for competitive or cooperative tasks?
• RQ5 Does the renderless mode enable efficient, long-duration training without performance degradation?

Key findings

• The agent successfully learned to navigate to a target location in approximately 1400 episodes, achieving convergence by episode 1000.
• The episodic reward curve showed a learning slowdown after 800 episodes, followed by stabilization and convergence, indicating effective policy learning.
• The use of TD3+HER significantly outperformed baseline DDPG, which failed to learn even basic navigation due to overestimation issues.
• The platform enabled training with renderless mode, eliminating graphical overhead and allowing server-side deployment and scalable distributed training.
• The simulation environment successfully supported multi-agent configurations, enabling competitive and cooperative task development such as dogfighting.
• The reward function based on distance to goal, combined with sparse terminal rewards, effectively guided the agent toward the goal without requiring dense shaping.