[Paper Review] DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking

Qing Lian, Tai Wang+2 more|arXiv (Cornell University)|Mar 29, 2023

Advanced Vision and Imaging4 citations

TL;DR

DORT proposes a recurrent framework that models dynamic object motion in multi-camera 3D object detection to correct localization bias from static assumptions. By using object-wise local 3D volumes and iterative refinement of motion and location, it enables precise temporal feature aggregation, achieving state-of-the-art 62.5% NDS and 57.6% AMOTA on nuScenes without bells and whistles.

ABSTRACT

Recent multi-camera 3D object detectors usually leverage temporal information to construct multi-view stereo that alleviates the ill-posed depth estimation. However, they typically assume all the objects are static and directly aggregate features across frames. This work begins with a theoretical and empirical analysis to reveal that ignoring the motion of moving objects can result in serious localization bias. Therefore, we propose to model Dynamic Objects in RecurrenT (DORT) to tackle this problem. In contrast to previous global Bird-Eye-View (BEV) methods, DORT extracts object-wise local volumes for motion estimation that also alleviates the heavy computational burden. By iteratively refining the estimated object motion and location, the preceding features can be precisely aggregated to the current frame to mitigate the aforementioned adverse effects. The simple framework has two significant appealing properties. It is flexible and practical that can be plugged into most camera-based 3D object detectors. As there are predictions of object motion in the loop, it can easily track objects across frames according to their nearest center distances. Without bells and whistles, DORT outperforms all the previous methods on the nuScenes detection and tracking benchmarks with 62.5\% NDS and 57.6\% AMOTA, respectively. The source code will be released.

Motivation & Objective

• Address the critical issue of localization bias in multi-camera 3D detection caused by assuming all objects are static during temporal feature aggregation.
• Overcome the chicken-and-egg problem of jointly estimating object motion and location by introducing a recurrent refinement mechanism.
• Reduce computational cost compared to global Bird's-Eye-View (BEV) representations while maintaining object-wise feature representation for motion modeling.
• Enable end-to-end joint 3D object detection and tracking by leveraging predicted motion to align detections across frames.
• Achieve state-of-the-art performance on nuScenes benchmark with minimal architectural modifications, demonstrating practical plug-in compatibility.

Proposed method

• Introduce object-wise local 3D volumes to extract motion-aware features, reducing computational cost compared to global BEV operations.
• Construct temporal cost volumes by warping local volumes from preceding frames to the current frame based on predicted object motion.
• Implement a recurrent module that iteratively refines object motion and location predictions using the warped cost volumes as supervision.
• Use a differentiable refinement loop where motion and location predictions are updated in tandem, enabling joint optimization.
• Leverage predicted motion to align detection results across frames, naturally enabling multi-object tracking without additional head design.
• Support multi-frame temporal modeling by extending the recurrent pipeline to incorporate more than two frames, improving motion estimation stability.

Figure 1 : Visualization of object localization from temporal correspondence. Previous work ignores the motion of moving objects, which leads to imprecise localization. Our work progressively refines the object’s location and motion so that the preceding features can be precisely aggregated.

Experimental results

Research questions

• RQ1 How does assuming static objects during temporal feature aggregation affect 3D object localization accuracy in multi-camera 3D detection?
• RQ2 Can iterative refinement of object motion and location improve temporal feature alignment and reduce depth estimation bias?
• RQ3 What is the impact of using object-wise local 3D volumes versus global BEV representations on computational efficiency and detection performance?
• RQ4 To what extent does explicit motion modeling enhance joint detection and tracking performance compared to static assumptions?
• RQ5 Can the proposed framework be flexibly integrated into existing camera-based 3D detectors without architectural overhaul?

Key findings

• DORT achieves 62.5% NDS on the nuScenes detection benchmark, outperforming all prior methods, including those with complex designs.
• The model achieves 57.6% AMOTA on the nuScenes tracking benchmark, demonstrating strong joint detection and tracking performance.
• Using ground truth motion improves mAP by 4.3% over the static assumption, proving the necessity of motion modeling for accurate correspondence.
• Predicted motion still yields a 2.9% mAP gain over static assumption, validating the effectiveness of the proposed dynamic modeling module.
• The ablation study confirms that object-wise local volumes achieve comparable performance to global BEV with significantly lower FLOPS, especially in the perspective view.
• Iterative refinement leads to progressive performance gains in motion and detection accuracy, with improvements continuing over multiple recurrent steps.

Figure 2 : Empirical analysis of the depth bias on the nuScenes dataset if objects are assumed static.