[Paper Review] Learning Unmanned Aerial Vehicle Control for Autonomous Target Following

Siyi Li, Tianbo Liu+3 more|arXiv (Cornell University)|Sep 24, 2017

Reinforcement Learning in Robotics27 references3 citations

TL;DR

This paper proposes a hierarchical deep reinforcement learning approach that combines a convolutional neural network policy with a PID controller to enable stable, data-efficient training for autonomous UAV target following. By learning high-level reference commands instead of low-level motor actions, the method achieves robust policy transfer from simulation to real-world DJI quadrotors with strong generalization and minimal catastrophic failures.

ABSTRACT

While deep reinforcement learning (RL) methods have achieved unprecedented successes in a range of challenging problems, their applicability has been mainly limited to simulation or game domains due to the high sample complexity of the trial-and-error learning process. However, real-world robotic applications often need a data-efficient learning process with safety-critical constraints. In this paper, we consider the challenging problem of learning unmanned aerial vehicle (UAV) control for tracking a moving target. To acquire a strategy that combines perception and control, we represent the policy by a convolutional neural network. We develop a hierarchical approach that combines a model-free policy gradient method with a conventional feedback proportional-integral-derivative (PID) controller to enable stable learning without catastrophic failure. The neural network is trained by a combination of supervised learning from raw images and reinforcement learning from games of self-play. We show that the proposed approach can learn a target following policy in a simulator efficiently and the learned behavior can be successfully transferred to the DJI quadrotor platform for real-world UAV control.

Motivation & Objective

• Address the high sample complexity and instability of model-free deep reinforcement learning in real-world UAV control.
• Overcome the limitations of traditional two-stage pipelines relying on manually tuned controllers and expert-labeled data.
• Enable end-to-end learning of perception and control policies that generalize across diverse environments.
• Achieve stable, safe training by integrating a conventional PID controller with a deep RL policy network.
• Demonstrate successful sim-to-real transfer of a learned policy to a physical DJI quadrotor platform.

Proposed method

• Represent the control policy as a single convolutional neural network (CNN) that maps raw image observations to high-level reference commands for a PID controller.
• Decompose training into supervised pre-training on raw images followed by reinforcement learning via self-play games to stabilize policy optimization.
• Use a hierarchical control architecture where the deep RL policy generates setpoints for a fixed PID controller, ensuring system stability during learning.
• Train the entire network using policy gradient methods with a scalar reward signal based on target tracking performance.
• Leverage domain randomization and self-play in simulation to improve robustness and generalization across unseen environments.
• Deploy the trained policy on a real DJI Matrice 100 quadrotor via ROS, with inference on a ground GPU and real-time command mapping via NUC.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning policy be trained stably in simulation for UAV target following without catastrophic failures during exploration?
• RQ2 How well can a policy trained in simulation generalize to real-world environments with sensor noise, control delays, and hardware differences?
• RQ3 To what extent does combining a learned policy with a conventional PID controller improve data efficiency and training stability?
• RQ4 Can end-to-end training of a perception-control policy achieve better generalization than separate perception and control modules?
• RQ5 What is the performance of a policy trained in simulation when directly transferred to a real-world UAV without fine-tuning?

Key findings

• The proposed hierarchical method achieved stable training without catastrophic failures, unlike standard model-free RL in the same setup.
• The end-to-end trained policy maintained the target within a small range of the goal state for over 5000 time steps in simulation, with average deviation below 0.1 in normalized state space.
• The policy generalized well to unseen scenarios with different backgrounds and distractors, demonstrating moderate robustness to visual variations.
• In real-world testing, the policy successfully followed a moving target for up to 4000 time steps (~3 minutes) despite sensor noise, IMU/GPS delays, and hardware discrepancies.
• The policy showed only a slight decrease in control precision in the real world compared to simulation, indicating strong sim-to-real transferability.
• The method achieved effective generalization even under partial occlusion, though performance degraded slightly when distractors were visually similar to the target.