[Paper Review] Aggressive Quadrotor Flight Using Curiosity-Driven Reinforcement Learning

Qiyu Sun, Jinbao Fang+2 more|arXiv (Cornell University)|Mar 26, 2022

Guidance and Control Systems4 citations

TL;DR

This paper proposes a curiosity-driven deep reinforcement learning method with a similarity-based curiosity module and branch structure exploration (BSE) to enable end-to-end aggressive quadrotor flight without predefined trajectories. The approach accelerates training, improves policy robustness, and achieves strong sim-to-real transferability, outperforming both baseline RL and trajectory-planning-based methods in simulation and real-world experiments on slalom and narrow-window missions.

ABSTRACT

The ability to perform aggressive movements, which are called aggressive flights, is important for quadrotors during navigation. However, aggressive quadrotor flights are still a great challenge to practical applications. The existing solutions to aggressive flights heavily rely on a predefined trajectory, which is a time-consuming preprocessing step. To avoid such path planning, we propose a curiosity-driven reinforcement learning method for aggressive flight missions and a similarity-based curiosity module is introduced to speed up the training procedure. A branch structure exploration (BSE) strategy is also applied to guarantee the robustness of the policy and to ensure the policy trained in simulations can be performed in real-world experiments directly. The experimental results in simulations demonstrate that our reinforcement learning algorithm performs well in aggressive flight tasks, speeds up the convergence process and improves the robustness of the policy. Besides, our algorithm shows a satisfactory simulated to real transferability and performs well in real-world experiments.

Motivation & Objective

• To eliminate the need for time-consuming trajectory planning in aggressive quadrotor flight missions.
• To address the sparse reward problem in reinforcement learning for underactuated, high-speed quadrotor control.
• To improve training efficiency and policy robustness through a similarity-based curiosity module.
• To enhance sim-to-real transferability using the branch structure exploration (BSE) strategy.
• To enable direct deployment of simulation-trained policies in real-world aggressive flight tasks.

Proposed method

• A similarity-based curiosity module computes intrinsic rewards based on state (position and velocity) similarity to encourage exploration in sparse reward environments.
• The method integrates the curiosity module with the TD3 algorithm to form a curiosity-driven TD3 agent for end-to-end control policy learning.
• The BSE (branch structure exploration) strategy enriches training diversity by enabling multiple action branches, improving policy generalization and robustness.
• Obstacles with varying positions and attitudes are randomly sampled during training to enhance environmental adaptability.
• The policy is trained exclusively in simulation and directly deployed in real-world experiments without fine-tuning.
• State observations include quadrotor position, velocity, and obstacle geometry, enabling real-time decision-making.

Experimental results

Research questions

• RQ1 Can a curiosity-driven reinforcement learning approach effectively train aggressive quadrotor flight policies without predefined trajectories?
• RQ2 How does a similarity-based curiosity module improve sample efficiency and convergence speed in sparse-reward aggressive flight tasks?
• RQ3 To what extent does the BSE strategy enhance the robustness and sim-to-real transferability of the trained policy?
• RQ4 How does the proposed method compare in performance and transferability to trajectory-planning-based approaches and standard RL baselines?
• RQ5 Can the policy trained in simulation be directly deployed on a real quadrotor for aggressive flight tasks?

Key findings

• Our method achieved a 0.044 average position error and 2.06° average rotation error in the slalom path mission, outperforming the nonlinear tracking controller (0.057 and 3.17°) and TD3 baseline (0.456 and 5.31°).
• In the narrow window mission, our method achieved a 0.187 m position error and 6.02° rotation error, significantly better than the TD3 baseline (0.431 m and 8.81°).
• The sim-to-real position difference was 0.12 m and angular difference 3.8°, demonstrating strong transferability despite simulation-real domain gaps.
• The combination of curiosity and BSE reduced sim-to-real trajectory discrepancies by 30% compared to standard TD3, indicating improved robustness.
• Ablation studies confirmed that both curiosity and BSE independently improved performance and transferability, with the full method achieving the best results.
• The policy trained in simulation successfully executed aggressive flight tasks in real-world environments without retraining, validating direct sim-to-real deployment.