[Paper Review] End-to-end Active Object Tracking and Its Real-world Deployment via Reinforcement Learning

Wenhan Luo, Peng Sun+4 more|arXiv (Cornell University)|Aug 10, 2018

Video Surveillance and Tracking Methods60 references6 citations

TL;DR

This paper proposes an end-to-end active object tracking system using deep reinforcement learning, where a ConvNet-LSTM agent directly predicts camera control actions from raw video frames in a simulator. The method achieves robust tracking generalization across unseen object paths, appearances, and backgrounds, and successfully transfers from simulation to real-world deployment on a TurtleBot robot with minimal real-world tuning.

ABSTRACT

We study active object tracking, where a tracker takes visual observations (i.e., frame sequences) as input and produces the corresponding camera control signals as output (e.g., move forward, turn left, etc.). Conventional methods tackle tracking and camera control tasks separately, and the resulting system is difficult to tune jointly. These methods also require significant human efforts for image labeling and expensive trial-and-error system tuning in the real world. To address these issues, we propose, in this paper, an end-to-end solution via deep reinforcement learning. A ConvNet-LSTM function approximator is adopted for the direct frame-to-action prediction. We further propose an environment augmentation technique and a customized reward function, which are crucial for successful training. The tracker trained in simulators (ViZDoom and Unreal Engine) demonstrates good generalization behaviors in the case of unseen object moving paths, unseen object appearances, unseen backgrounds, and distracting objects. The system is robust and can restore tracking after occasional lost of the target being tracked. We also find that the tracking ability, obtained solely from simulators, can potentially transfer to real-world scenarios. We demonstrate successful examples of such transfer, via experiments over the VOT dataset and the deployment of a real-world robot using the proposed active tracker trained in simulation.

Motivation & Objective

• To address the limitations of conventional passive trackers that separate tracking and camera control, requiring manual tuning and extensive real-world trial-and-error.
• To reduce human effort in labeling and system tuning by training an end-to-end agent in simulated environments.
• To enable robust generalization of active tracking to unseen object trajectories, appearances, backgrounds, and distractors.
• To bridge the sim-to-real gap through advanced environment augmentation and appropriate action space design for real-world robot deployment.
• To demonstrate successful real-world deployment of a tracker trained exclusively in simulation on a physical TurtleBot.

Proposed method

• A ConvNet-LSTM architecture is used to map raw video frames directly to camera control actions (e.g., move forward, turn left), enabling end-to-end learning.
• The A3C reinforcement learning algorithm is employed to train the agent using a customized dense reward function that encourages target localization in the image center and size consistency.
• Environment augmentation is applied by dynamically varying object appearance, background, trajectory, and distractors in the simulator to improve generalization.
• The action space is carefully designed—discrete or continuous—to better match real-world robot kinematics and improve sim-to-real transfer.
• The virtual environment is enhanced via custom APIs and plugins to simulate diverse real-world conditions during training.
• The system is evaluated both in simulation and on real-world robot deployment, with transfer validated on VOT dataset clips and physical robot trials.

Experimental results

Research questions

• RQ1 Can an end-to-end active tracking policy trained solely in simulation generalize to unseen object motion paths and appearances?
• RQ2 How effective is reinforcement learning with custom reward functions in enabling robust tracking under distractors and background variations?
• RQ3 To what extent can a policy trained in a simulated environment successfully transfer to real-world robot control without real-world fine-tuning?
• RQ4 What role does environment augmentation play in improving the generalization and robustness of the active tracker?
• RQ5 How does the choice of action space (discrete vs. continuous) affect tracking performance and real-world deployment success?

Key findings

• The tracker generalizes effectively to unseen object trajectories, appearances, backgrounds, and distractors, maintaining stable tracking even after temporary target loss.
• The system achieves a success rate of 0.90 in the indoor discrete action setting and 0.70 in the outdoor continuous action setting, demonstrating strong performance across diverse conditions.
• The tracker successfully transfers from simulation to real-world deployment on a TurtleBot, achieving stable active tracking in both indoor and outdoor environments.
• The agent learns to maintain the target near the image center and preserve consistent target size, indicating effective motion-aware control decisions.
• The use of continuous action space improves performance in outdoor settings, with a success rate of 0.70 compared to 0.90 in indoor discrete settings.
• Qualitative results show the tracker avoids naive 'stop' actions and instead follows the target’s motion direction, indicating learned motion prediction through the LSTM encoder.