[Paper Review] RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network

Zhiwei Lin, Zhe Liu+3 more|arXiv (Cornell University)|Sep 8, 2024

Infrared Target Detection Methodologies4 citations

TL;DR

This paper proposes RCBEVDet++, a novel radar-camera fusion 3D perception framework that enhances multi-view camera-based 3D detection via a specialized radar feature extractor, RadarBEVNet, and a cross-attention fusion module (CAMF) with sparse fusion. It achieves state-of-the-art performance on nuScenes, attaining 72.73 NDS and 67.34 mAP in 3D object detection using ViT-L backbone without test-time augmentation or ensembling.

ABSTRACT

Perceiving the surrounding environment is a fundamental task in autonomous driving. To obtain highly accurate perception results, modern autonomous driving systems typically employ multi-modal sensors to collect comprehensive environmental data. Among these, the radar-camera multi-modal perception system is especially favored for its excellent sensing capabilities and cost-effectiveness. However, the substantial modality differences between radar and camera sensors pose challenges in fusing information. To address this problem, this paper presents RCBEVDet, a radar-camera fusion 3D object detection framework. Specifically, RCBEVDet is developed from an existing camera-based 3D object detector, supplemented by a specially designed radar feature extractor, RadarBEVNet, and a Cross-Attention Multi-layer Fusion (CAMF) module. Firstly, RadarBEVNet encodes sparse radar points into a dense bird's-eye-view (BEV) feature using a dual-stream radar backbone and a Radar Cross Section aware BEV encoder. Secondly, the CAMF module utilizes a deformable attention mechanism to align radar and camera BEV features and adopts channel and spatial fusion layers to fuse them. To further enhance RCBEVDet's capabilities, we introduce RCBEVDet++, which advances the CAMF through sparse fusion, supports query-based multi-view camera perception models, and adapts to a broader range of perception tasks. Extensive experiments on the nuScenes show that our method integrates seamlessly with existing camera-based 3D perception models and improves their performance across various perception tasks. Furthermore, our method achieves state-of-the-art radar-camera fusion results in 3D object detection, BEV semantic segmentation, and 3D multi-object tracking tasks. Notably, with ViT-L as the image backbone, RCBEVDet++ achieves 72.73 NDS and 67.34 mAP in 3D object detection without test-time augmentation or model ensembling.

Motivation & Objective

• To address the challenge of fusing millimeter-wave radar and multi-view camera data due to significant modality differences and spatial misalignment.
• To design a radar-specific feature extractor that accounts for unique radar characteristics such as Radar Cross Section (RCS) and sparse point clouds.
• To develop a robust, cross-attention-based fusion mechanism that aligns and fuses radar and camera features in Bird's Eye View (BEV) space.
• To extend the framework to support query-based multi-view detectors and broader perception tasks like BEV semantic segmentation and 3D multi-object tracking.
• To demonstrate generalization across diverse backbone architectures and detector designs in real-world autonomous driving scenarios.

Proposed method

• RadarBEVNet uses a dual-stream encoder to extract point-based and transformer-based representations from sparse radar points.
• An Injection and Extraction module fuses the dual-stream representations before projecting them into BEV space using RCS-aware scattering to model object size and distribution.
• The CAMF module employs deformable cross-attention to align radar and camera BEV features, correcting spatial misalignment.
• Channel and spatial fusion layers consolidate multi-modal features for improved detection performance.
• RCBEVDet++ enhances CAMF with sparse fusion to support query-based detection heads and enables broader task adaptability.
• The framework is plug-and-play, compatible with existing camera-based 3D perception models, including LSS-based and transformer-based detectors.

Figure 1: Overall pipeline of RCBEVDet. Firstly, multi-view images are encoded and transformed into image BEV features. Concurrently, radar point clouds are processed by the proposed RadarBEVNet to extract radar BEV features. Subsequently, features from both radar and cameras are dynamically aligned

Experimental results

Research questions

• RQ1 How can radar and camera features be effectively aligned in BEV space despite their inherent modality and spatial differences?
• RQ2 To what extent can a radar-specific feature extractor that incorporates RCS awareness improve 3D perception performance compared to LiDAR-inspired encoders?
• RQ3 Can a cross-attention fusion mechanism outperform simple concatenation or summation in radar-camera BEV fusion?
• RQ4 How does the proposed method generalize across different backbone architectures (CNN and Transformer) and detector designs (LSS and transformer-based)?
• RQ5 Can the framework be extended to support multiple perception tasks beyond 3D object detection, such as BEV semantic segmentation and 3D multi-object tracking?

Key findings

• RCBEVDet++ achieves 72.73 NDS and 67.34 mAP in 3D object detection on nuScenes using ViT-L as the image backbone, without test-time augmentation or model ensembling.
• The method improves baseline performance by 4.9 NDS and 4.8–10.2 mAP across various backbone architectures, including ResNet-18, ResNet-50, Swin-T, and DLA34.
• For LSS-based detectors, RCBEVDet boosts BEVDet by 5.6 NDS and BEVDepth by 4.9 NDS.
• For transformer-based detectors, RCBEVDet++ achieves 5.6–5.9 NDS improvements, with mAP gains of 8.3–8.7, indicating stronger benefits for implicit 3D position learning.
• The framework generalizes well across detector architectures, improving all tested models including StreamPETR and SparseBEV.
• RCBEVDet++ achieves new state-of-the-art results in 3D object detection, BEV semantic segmentation, and 3D multi-object tracking on the nuScenes benchmark.

(a)