[论文解读] Joint Monocular 3D Vehicle Detection and Tracking

Hou-Ning Hu, Qi-Zhi Cai+6 more|arXiv (Cornell University)|Nov 26, 2018

Video Surveillance and Tracking Methods参考文献 60被引用 49

一句话总结

本论文提出一个在线单目框架，结合深度感知数据关联、具有LSTM的三维运动建模以及遮挡处理，在 GTA 基于的合成数据、KITTI 和 Argoverse 上进行了验证。

ABSTRACT

Vehicle 3D extents and trajectories are critical cues for predicting the future location of vehicles and planning future agent ego-motion based on those predictions. In this paper, we propose a novel online framework for 3D vehicle detection and tracking from monocular videos. The framework can not only associate detections of vehicles in motion over time, but also estimate their complete 3D bounding box information from a sequence of 2D images captured on a moving platform. Our method leverages 3D box depth-ordering matching for robust instance association and utilizes 3D trajectory prediction for re-identification of occluded vehicles. We also design a motion learning module based on an LSTM for more accurate long-term motion extrapolation. Our experiments on simulation, KITTI, and Argoverse datasets show that our 3D tracking pipeline offers robust data association and tracking. On Argoverse, our image-based method is significantly better for tracking 3D vehicles within 30 meters than the LiDAR-centric baseline methods.

研究动机与目标

• Motivate and enable 3D vehicle detection and tracking from monocular video without LiDAR or stereo inputs.
• Develop an online framework that jointly detects 3D vehicle layouts and links them across frames.
• Leverage depth-ordering and occlusion-aware data association to improve tracking robustness in ego-motion scenarios.
• Introduce an LSTM-based motion model to extrapolate 3D vehicle trajectories over time.
• Create a synthetic GTA-based dataset with ground-truth 3D trajectories to support data-hungry learning for 3D tracking.

提出的方法

• Detect 2D proposals with Faster R-CNN and regress a 3D center projection for each object.
• Estimate full 3D box information (P, O, D, depth, and 3D center projection) from ROI features using a CNN sub-network.
• Track objects online by forming a 3D trajectory in world coordinates and using depth-ordering matching and occlusion-aware association for data association.
• Model 3D motion with two LSTMs: a Prediction LSTM (P-LSTM) for velocity and position and an Updating LSTM (U-LSTM) to refine location and velocity.
• Fuse single-frame 3D estimates over time to refine 3D bounding boxes and trajectories, factoring ego-motion with camera transform.

实验结果

研究问题

• RQ1 Can monocular video provide reliable 3D vehicle bounding boxes and trajectories when combined with ego-motion sensors?
• RQ2 Does depth-aware data association improve cross-frame object identity preservation under occlusion and ego-motion?
• RQ3 Can an LSTM-based motion model outperform Kalman-filter-based smoothing for 3D vehicle trajectories in monocular settings?
• RQ4 How much does projecting the 3D center into the image improve tracking accuracy and ID reliability?
• RQ5 What is the impact of training data scale on 3D estimation and tracking performance in synthetic versus real-world datasets?

主要发现

• The proposed framework achieves robust 3D detection and tracking from monocular video with occlusion-aware association and depth-ordering, reducing mismatches by 6-8% in ablation experiments.
• An LSTM-based motion model outperforms single-frame estimation and 3D Kalman filtering in 3D IoU tracking accuracy across IoU thresholds.
• Projecting the 3D center projection into the image significantly lowers ID switches and track fragmentation compared to using the 2D center.
• Depth-order matching improves data association robustness to ego-motion and occlusion, enhancing MOTA/MOTP metrics in end-to-end evaluations.
• Larger synthetic GTA-based training data yields consistent gains in depth estimation accuracy and 3D layout quality, highlighting data-hungry model benefits.