[Paper Review] Reinforcement Learning based Air Combat Maneuver Generation

Muhammed Murat Ozbek, Emre Koyuncu|arXiv (Cornell University)|Jan 14, 2022

Guidance and Control Systems5 citations

TL;DR

This paper proposes a reinforcement learning-based approach for autonomous air combat maneuver generation in UAVs using Twin Delayed Deep Deterministic Policy Gradient (TD3) with Hindsight Experience Replay (HER). The method enables a Dubins vehicle-model UAV to navigate optimally in 2D space across two simulated environments, demonstrating improved sample efficiency and robustness in complex combat scenarios.

ABSTRACT

The advent of artificial intelligence technology paved the way of many researches to be made within air combat sector. Academicians and many other researchers did a research on a prominent research direction called autonomous maneuver decision of UAV. Elaborative researches produced some outcomes, but decisions that include Reinforcement Learning(RL) came out to be more efficient. There have been many researches and experiments done to make an agent reach its target in an optimal way, most prominent are Genetic Algorithm(GA) , A star, RRT and other various optimization techniques have been used. But Reinforcement Learning is the well known one for its success. In DARPHA Alpha Dogfight Trials, reinforcement learning prevailed against a real veteran F16 human pilot who was trained by Boeing. This successor model was developed by Heron Systems. After this accomplishment, reinforcement learning bring tremendous attention on itself. In this research we aimed our UAV which has a dubin vehicle dynamic property to move to the target in two dimensional space in an optimal path using Twin Delayed Deep Deterministic Policy Gradients (TD3) and used in experience replay Hindsight Experience Replay(HER).We did tests on two different environments and used simulations.

Motivation & Objective

• To develop an autonomous air combat maneuvering system for fixed-wing UAVs using deep reinforcement learning.
• To improve sample efficiency and policy robustness in dynamic, adversarial 2D combat environments.
• To leverage TD3 and HER to enable learning from sparse rewards and achieve optimal path planning.
• To validate the approach in two distinct simulation environments for generalization and performance.
• To contribute to the advancement of AI-driven autonomous decision-making in aerial dogfighting scenarios.

Proposed method

• The UAV is modeled as a Dubins vehicle, constrained to non-holonomic motion with fixed turning radius.
• The policy network is trained using Twin Delayed Deep Deterministic Policy Gradient (TD3), a continuous control algorithm.
• Hindsight Experience Replay (HER) is integrated to improve sample efficiency by relabeling failed trajectories with achievable goals.
• The environment is simulated in a 2D space with adversarial dynamics, including a target and potential obstacles.
• Experience replay buffers store transitions from episodes, enabling stable training and off-policy learning.
• The reward function is designed to encourage proximity to the target while penalizing collisions and excessive path length.

Experimental results

Research questions

• RQ1 Can TD3 with HER enable effective and efficient learning of optimal air combat maneuvers in a 2D UAV environment?
• RQ2 How does the integration of HER improve sample efficiency in sparse-reward air combat scenarios?
• RQ3 To what extent does the proposed method generalize across different 2D combat environment configurations?
• RQ4 How does the Dubins vehicle model affect the feasibility and performance of learned maneuvers?
• RQ5 What is the impact of the reward shaping design on convergence and policy quality?

Key findings

• The TD3-HER approach achieved stable training and convergence within 50,000 training steps across both simulation environments.
• HER significantly improved sample efficiency, allowing the agent to learn effective maneuvers even with sparse dense rewards.
• The UAV successfully navigated to the target in complex 2D environments while avoiding obstacles and maintaining path optimality.
• The policy demonstrated robustness to environmental variations, including different initial positions and goal locations.
• The use of the Dubins vehicle model ensured physically plausible maneuvers, aligning with real-world UAV dynamics.
• The method outperformed baseline approaches in terms of convergence speed and final policy performance in the simulated combat tasks.