[Paper Review] Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

Wenjie Luo, Bin Yang+1 more|arXiv (Cornell University)|Dec 22, 2020

Advanced Vision and Imaging26 references348 citations

TL;DR

FaF proposes an end-to-end single 3D CNN that jointly detects, tracks, and forecasts motion from BEV voxelized point clouds, running in as fast as 30 ms and outperforming prior methods.

ABSTRACT

In this paper we propose a novel deep neural network that is able to jointly reason about 3D detection, tracking and motion forecasting given data captured by a 3D sensor. By jointly reasoning about these tasks, our holistic approach is more robust to occlusion as well as sparse data at range. Our approach performs 3D convolutions across space and time over a bird's eye view representation of the 3D world, which is very efficient in terms of both memory and computation. Our experiments on a new very large scale dataset captured in several north american cities, show that we can outperform the state-of-the-art by a large margin. Importantly, by sharing computation we can perform all tasks in as little as 30 ms.

Motivation & Objective

• Motivate a holistic approach to 3D detection, tracking, and motion forecasting to improve robustness to occlusion and sparse data.
• Develop a single-stage, end-to-end network that exploits temporal BEV representations for joint tasks.
• Show that shared computation across tasks yields real-time performance and improved accuracy.

Proposed method

• Represent 3D LiDAR data as a BEV voxel grid and apply 2D convolutions across height as channels for single-frame processing to avoid sparsity waste.
• Extend to multiple frames by stacking temporal voxel grids into a 4D tensor and compare early fusion and late fusion temporal aggregation schemes.
• Use SSD-style multi-box predictions with predefined anchors in BEV, predicting box location, size, and heading (sine/cos) with a smooth L1 loss.
• Decode tracklets by aggregating current detections with past future predictions to form robust track IDs through occlusions.
• Train with a joint loss combining classification and regression terms across current and future frames (n-frame forecasting).
• Evaluate detection, tracking, and motion forecasting jointly on a large-scale, city-scale LiDAR dataset.

Experimental results

Research questions

• RQ1 Can a single end-to-end 3D CNN jointly detect, track, and forecast motion from multi-frame LiDAR BEV representations?
• RQ2 Does sharing computations across detection, tracking, and forecasting improve robustness to occlusion and long-range sparsity while meeting real-time latency targets?
• RQ3 How do early fusion and late fusion temporal strategies compare in terms of accuracy and efficiency?
• RQ4 What are the quantitative gains in detection mAP, tracking metrics, and short-term forecasting error when usingFaF compared to state-of-the-art baselines?

Key findings

• FaF achieves higher detection mAP than 2D detectors adapted to 3D with comparable or faster latency, outperforming baselines at IoU 0.7.
• Ablation shows late fusion yields +1.4% mAP over early fusion, and adding future-frame prediction yields about +6 percentage points in mAP at IoU 0.7 over a single-frame detector.
• Tracking performance improves versus a Hungarian baseline by 6% in MOTA and increases Mostly-Tracked (MT) by 20%.
• Motion forecasting achieves average L2 error under 0.33 meters for 10-frame-ahead predictions.
• FaF runs in as little as 30 ms, enabling real-time joint detection, tracking, and forecasting in autonomous driving.