[Paper Review] Data-Efficient Deep Reinforcement Learning for Attitude Control of Fixed-Wing UAVs: Field Experiments

Eivind Bøhn, Erlend M. Coates+2 more|arXiv (Cornell University)|Nov 7, 2021

Adaptive Control of Nonlinear Systems4 citations

TL;DR

This paper presents a data-efficient deep reinforcement learning (DRL) approach for attitude control of fixed-wing UAVs, training a controller in simulation with as little as three minutes of real flight data and transferring it directly to real-world flight without online fine-tuning. The method achieves performance comparable to the state-of-the-art ArduPlane PID controller by leveraging sim-to-real techniques, including actuation delay, dynamic diversity, and smoothness regularization, resulting in stable, low-oscillation control responses.

ABSTRACT

Attitude control of fixed-wing unmanned aerial vehicles (UAVs) is a difficult control problem in part due to uncertain nonlinear dynamics, actuator constraints, and coupled longitudinal and lateral motions. Current state-of-the-art autopilots are based on linear control and are thus limited in their effectiveness and performance. Deep reinforcement learning (DRL) is a machine learning method to automatically discover optimal control laws through interaction with the controlled system, which can handle complex nonlinear dynamics. We show in this paper that DRL can successfully learn to perform attitude control of a fixed-wing UAV operating directly on the original nonlinear dynamics, requiring as little as three minutes of flight data. We initially train our model in a simulation environment and then deploy the learned controller on the UAV in flight tests, demonstrating comparable performance to the state-of-the-art ArduPlane proportional-integral-derivative (PID) attitude controller with no further online learning required. Learning with significant actuation delay and diversified simulated dynamics were found to be crucial for successful transfer to control of the real UAV. In addition to a qualitative comparison with the ArduPlane autopilot, we present a quantitative assessment based on linear analysis to better understand the learning controller's behavior.

Motivation & Objective

• To develop a data-efficient deep reinforcement learning controller for fixed-wing UAV attitude control that minimizes reliance on extensive real-world data.
• To address the sim-to-real gap in UAV control by robustifying the learned policy through simulated actuation delay and dynamic variation.
• To enable direct deployment of a DRL controller on real UAV hardware without online fine-tuning or retraining.
• To achieve performance comparable to state-of-the-art linear PID controllers while handling nonlinear, coupled, and underactuated dynamics.

Proposed method

• The controller is trained using the Soft Actor-Critic (SAC) algorithm in a high-fidelity simulation environment that models the nonlinear dynamics of the Skywalker X8 fixed-wing UAV.
• The state representation includes a convolutional input layer that processes a sequence of past states, enhancing temporal awareness and improving control smoothness.
• The training environment introduces significant actuation delay and diverse aerodynamic parameters to simulate real-world uncertainties and improve robustness.
• Sparse dense rewards are used, with additional terms to penalize aggressive control actions and promote temporal and spatial smoothness in control signals.
• The policy is trained exclusively on simulated data, with no real-time interaction during learning, and deployed directly on the real UAV.
• The method includes sim-to-real transfer techniques such as domain randomization and latency injection to close the reality gap and ensure field deployability.

Experimental results

Research questions

• RQ1 Can a DRL controller trained solely on simulated data achieve stable and high-performance attitude control on a real fixed-wing UAV with minimal real-world data?
• RQ2 What specific sim-to-real techniques are most effective in enabling successful transfer of a DRL policy from simulation to real flight?
• RQ3 How does the inclusion of state history and convolutional processing in the input representation affect controller performance and smoothness?
• RQ4 To what extent can DRL outperform or match the performance of established linear PID controllers in nonlinear, coupled UAV dynamics?
• RQ5 What role does data efficiency play in reducing overfitting to the simulation model and improving real-world generalization?

Key findings

• The DRL controller achieved stable and comparable performance to the ArduPlane PID controller in real-world flight tests, with no need for online adaptation or fine-tuning.
• The controller demonstrated flightworthiness after only 10,000 training time steps (approximately 3 minutes of real flight data), indicating high data efficiency.
• The base model with a convolutional input layer reached convergence in around 40,000 time steps (13 minutes of real flight), with significantly lower proportional gains and 50% better smoothness metric than a fully connected baseline.
• The model with only a single past state (h=1) and fully connected input failed to stabilize the UAV, highlighting the importance of temporal state history.
• Incorporating actuation delay and dynamic variation during training was crucial for successful sim-to-real transfer and robustness.
• The controller generated smoother control signals and attitude responses than the FC baseline, indicating superior handling of system dynamics and reduced oscillation.