[论文解读] RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network

Zhiwei Lin, Zhe Liu+3 more|arXiv (Cornell University)|Sep 8, 2024

Infrared Target Detection Methodologies被引用 4

一句话总结

该论文提出RCBEVDet++，一种新颖的雷达-相机融合3D感知框架，通过专用的雷达特征提取器RadarBEVNet和基于稀疏融合的交叉注意力融合模块（CAMF），提升了多视角相机的3D检测性能。该方法在nuScenes数据集上实现了最先进性能，使用ViT-L主干网络时，3D目标检测的NDS达到72.73，mAP达到67.34，且无需测试时增强或模型集成。

ABSTRACT

Perceiving the surrounding environment is a fundamental task in autonomous driving. To obtain highly accurate perception results, modern autonomous driving systems typically employ multi-modal sensors to collect comprehensive environmental data. Among these, the radar-camera multi-modal perception system is especially favored for its excellent sensing capabilities and cost-effectiveness. However, the substantial modality differences between radar and camera sensors pose challenges in fusing information. To address this problem, this paper presents RCBEVDet, a radar-camera fusion 3D object detection framework. Specifically, RCBEVDet is developed from an existing camera-based 3D object detector, supplemented by a specially designed radar feature extractor, RadarBEVNet, and a Cross-Attention Multi-layer Fusion (CAMF) module. Firstly, RadarBEVNet encodes sparse radar points into a dense bird's-eye-view (BEV) feature using a dual-stream radar backbone and a Radar Cross Section aware BEV encoder. Secondly, the CAMF module utilizes a deformable attention mechanism to align radar and camera BEV features and adopts channel and spatial fusion layers to fuse them. To further enhance RCBEVDet's capabilities, we introduce RCBEVDet++, which advances the CAMF through sparse fusion, supports query-based multi-view camera perception models, and adapts to a broader range of perception tasks. Extensive experiments on the nuScenes show that our method integrates seamlessly with existing camera-based 3D perception models and improves their performance across various perception tasks. Furthermore, our method achieves state-of-the-art radar-camera fusion results in 3D object detection, BEV semantic segmentation, and 3D multi-object tracking tasks. Notably, with ViT-L as the image backbone, RCBEVDet++ achieves 72.73 NDS and 67.34 mAP in 3D object detection without test-time augmentation or model ensembling.

研究动机与目标

• 为解决因显著的模态差异和空间错位而导致的毫米波雷达与多视角相机数据融合挑战。
• 设计一种针对雷达的特征提取器，以考虑雷达特有的特性，如雷达散射截面（RCS）和稀疏点云。
• 开发一种稳健的基于交叉注意力的融合机制，实现在鸟瞰图（BEV）空间中对齐并融合雷达与相机特征。
• 将框架扩展以支持基于查询的多视角检测器，并支持更广泛的感知任务，如BEV语义分割和3D多目标跟踪。
• 在真实世界自动驾驶场景中，验证该方法在多种主干网络架构和检测器设计上的泛化能力。

提出的方法

• RadarBEVNet使用双流编码器，从稀疏雷达点中提取基于点的表示和基于Transformer的表示。
• 通过注入与提取模块融合双流表示，并利用RCS感知的散射方法将其投影到BEV空间，以建模目标大小与分布。
• CAMF模块采用可变形交叉注意力机制，对齐雷达与相机的BEV特征，纠正空间错位。
• 通道与空间融合层整合多模态特征，以提升检测性能。
• RCBEVDet++通过引入稀疏融合机制增强CAMF，以支持基于查询的检测头，并提升任务适应性。
• 该框架即插即用，兼容现有的基于相机的3D感知模型，包括LSS-based和基于Transformer的检测器。

Figure 1: Overall pipeline of RCBEVDet. Firstly, multi-view images are encoded and transformed into image BEV features. Concurrently, radar point clouds are processed by the proposed RadarBEVNet to extract radar BEV features. Subsequently, features from both radar and cameras are dynamically aligned

实验结果

研究问题

• RQ1 尽管存在固有的模态差异和空间差异，如何在BEV空间中有效对齐雷达与相机特征？
• RQ2 与基于LiDAR启发的编码器相比，融合RCS感知的雷达专用特征提取器能在多大程度上提升3D感知性能？
• RQ3 基于交叉注意力的融合机制是否能优于简单的拼接或求和操作？
• RQ4 所提出方法在不同主干网络架构（CNN与Transformer）及检测器设计（LSS与基于Transformer）上的泛化能力如何？
• RQ5 该框架能否扩展以支持3D目标检测之外的多种感知任务，如BEV语义分割和3D多目标跟踪？

主要发现

• RCBEVDet++在nuScenes数据集上使用ViT-L作为图像主干网络时，3D目标检测的NDS达到72.73，mAP达到67.34，且无需测试时增强或模型集成。
• 该方法在多种主干网络架构（包括ResNet-18、ResNet-50、Swin-T和DLA34）上，将基线性能提升4.9 NDS和4.8–10.2 mAP。
• 对于LSS-based检测器，RCBEVDet将BEVDet的性能提升5.6 NDS，将BEVDepth提升4.9 NDS。
• 对于基于Transformer的检测器，RCBEVDet++实现5.6–5.9 NDS的性能提升，mAP提升8.3–8.7，表明其在隐式3D位置学习方面具有更强的增益优势。
• 该框架在检测器架构间表现出良好的泛化能力，显著提升了所有测试模型，包括StreamPETR和SparseBEV。
• RCBEVDet++在nuScenes基准上实现了3D目标检测、BEV语义分割和3D多目标跟踪任务的新SOTA结果。

(a)