[Paper Review] Deep Tracking on the Move: Learning to Track the World from a Moving Vehicle using Recurrent Neural Networks

Julie Dequaire, Dushyant Rao+3 more|arXiv (Cornell University)|Sep 29, 2016

Autonomous Vehicle Technology and Safety14 references15 citations

TL;DR

This paper proposes an end-to-end deep learning framework using recurrent neural networks (RNNs) with Spatial Transformer modules to track dynamic and static objects in urban environments from a moving vehicle's perspective. By leveraging estimated egomotion, the model predicts unoccluded occupancy grids from raw laser scans and accurately tracks objects—including those in complete occlusion—outperforming baseline methods in future state prediction and robustness to motion-induced distortions.

ABSTRACT

This paper presents an end-to-end approach for tracking static and dynamic objects for an autonomous vehicle driving through crowded urban environments. Unlike traditional approaches to tracking, this method is learned end-to-end, and is able to directly predict a full unoccluded occupancy grid map from raw laser input data. Inspired by the recently presented DeepTracking approach [Ondruska, 2016], we employ a recurrent neural network (RNN) to capture the temporal evolution of the state of the environment, and propose to use Spatial Transformer modules to exploit estimates of the egomotion of the vehicle. Our results demonstrate the ability to track a range of objects, including cars, buses, pedestrians, and cyclists through occlusion, from both moving and stationary platforms, using a single learned model. Experimental results demonstrate that the model can also predict the future states of objects from current inputs, with greater accuracy than previous work.

Motivation & Objective

• To develop an end-to-end learning framework for tracking objects in dynamic urban environments from a moving autonomous vehicle.
• To overcome limitations of traditional multi-stage tracking pipelines that rely on hand-engineered components.
• To enable accurate prediction of object states, including future positions and occluded tracks, using only raw laser input.
• To integrate estimated egomotion into the network via Spatial Transformer modules to improve tracking robustness under vehicle motion.
• To demonstrate superior performance in tracking diverse objects—cars, pedestrians, cyclists, buses—through occlusion and over time.

Proposed method

• The model uses a gated recurrent unit (GRU)-based RNN to encode temporal evolution of the environment from sequential laser scans.
• Spatial Transformer modules are applied to transform the feature maps based on estimated egomotion, aligning the feature space to the world frame.
• The network is trained end-to-end to predict a full unoccluded occupancy grid directly from raw laser input, bypassing intermediate detection and association stages.
• Egomotion estimates are used as a proxy for true vehicle motion to correct for relative motion between sensor and environment.
• The architecture enables memory of object positions and velocities in the RNN hidden states, allowing future state prediction.
• A masking strategy during inference simulates occlusion by blacking out input frames every 5 frames, testing generalization to unseen occlusions.

Experimental results

Research questions

• RQ1 Can an end-to-end deep learning model track multiple object types through occlusion without explicit detection or data association?
• RQ2 How does incorporating estimated egomotion via Spatial Transformers improve tracking accuracy on a moving platform compared to a static model?
• RQ3 To what extent can the model predict future object states, including those behind occlusions, using only current input?
• RQ4 Does the model generalize to complex urban scenes with diverse dynamic and static objects under realistic vehicle motion?
• RQ5 How does the model’s performance compare to a baseline that ignores egomotion, especially in long-term occlusion?

Key findings

• The Spatial Transformer Model (STM) significantly outperforms the baseline DeepTracking model in predicting future object states, especially during and after complete occlusion.
• The STM maintains accurate tracking of two pedestrians occluded by a moving vehicle across five frames, while the baseline model fails to recover their positions.
• The baseline model achieves a surprisingly high F1 score despite ignoring egomotion, likely due to the dataset’s relatively constant vehicle velocity and high static object content.
• The STM achieves a clear improvement in F1 measure over the baseline across all future frames, demonstrating the value of egomotion integration.
• The model successfully tracks cars, buses, pedestrians, and cyclists through occlusion, even when objects are fully hidden for multiple frames.
• The system generalizes well to real-world urban driving conditions, including complex relative motions and dynamic scene changes.