[Paper Review] BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection

Junjie Huang, Guan Huang|arXiv (Cornell University)|Mar 31, 2022

Advanced Neural Network Applications156 citations

TL;DR

BEVDet4D extends BEVDet to a spatial-temporal 4D space by fusing current and previous frame BEV features, improving velocity prediction with negligible overhead and achieving state-of-the-art vision-based 3D detection on nuScenes.

ABSTRACT

Single frame data contains finite information which limits the performance of the existing vision-based multi-camera 3D object detection paradigms. For fundamentally pushing the performance boundary in this area, a novel paradigm dubbed BEVDet4D is proposed to lift the scalable BEVDet paradigm from the spatial-only 3D space to the spatial-temporal 4D space. We upgrade the naive BEVDet framework with a few modifications just for fusing the feature from the previous frame with the corresponding one in the current frame. In this way, with negligible additional computing budget, we enable BEVDet4D to access the temporal cues by querying and comparing the two candidate features. Beyond this, we simplify the task of velocity prediction by removing the factors of ego-motion and time in the learning target. As a result, BEVDet4D with robust generalization performance reduces the velocity error by up to -62.9%. This makes the vision-based methods, for the first time, become comparable with those relied on LiDAR or radar in this aspect. On challenge benchmark nuScenes, we report a new record of 54.5% NDS with the high-performance configuration dubbed BEVDet4D-Base, which surpasses the previous leading method BEVDet-Base by +7.3% NDS. The source code is publicly available for further research at https://github.com/HuangJunJie2017/BEVDet .

Motivation & Objective

• Extend BEVDet from spatial-only to spatial-temporal 4D fusion to leverage temporal cues.
• Retain BEVDet’s architecture while incorporating a lightweight temporal fusion mechanism.
• Simplify velocity learning by predicting positional offset between adjacent BEV features rather than absolute velocity.
• Demonstrate improved velocity, orientation, and attribute errors with minimal inference overhead on nuScenes.

Proposed method

• Retain BEVDet image-view encoder, view transformer, BEV encoder, and task heads; add temporal fusion by storing the previous BEV feature and concatenating it with the current frame after alignment.
• Apply a simple spatial alignment to previous-frame BEV features to remove ego-motion before fusion.
• Introduce an extra BEV encoder before temporal fusion to adjust sparse features and stabilize learning.
• Formulate velocity prediction as the translation between adjacent BEV features, removing ego-motion from the target learning signal.
• Explore alignment via rotation and translation (Eq. 2) and implement feature alignment (Eq. 3) with bilinear interpolation when needed.
• Evaluate with nuScenes metrics (mAP, mATE, mASE, mAOE, mAVE, mAAE, NDS) and report inference speed (FPS).

Experimental results

Research questions

• RQ1 Can temporal fusion of BEV features from two adjacent frames improve velocity and overall 3D object detection performance in a purely vision-based, multi-camera setup?
• RQ2 What alignment and network adjustments are necessary to decouple ego-motion from temporal feature differences and stabilize learning for velocity prediction?
• RQ3 How does BEVDet4D compare to state-of-the-art vision-based baselines on nuScenes in terms of accuracy and speed?

Key findings

• BEVDet4D-Tiny reduces velocity error by 62.9% (AVE from 0.909 to 0.337 mAVE) and improves NDS by 8.4% over BEVDet-Tiny on nuScenes val.
• BEVDet4D-Base achieves 54.5% NDS on nuScenes val and 56.9% NDS on the test set, surpassing prior vision-based methods and BEVDet variants while maintaining similar latency.
• Temporal fusion after an extra BEV encoder yields best trade-offs, with notable gains in mAP, NDS, and velocity metrics compared to earlier configurations.
• Using temporal cues, BEVDet4D narrows the gap to LiDAR/radar-based velocity precision, achieving competitive AVE to non-RGB modalities on nuScenes validation.