[Paper Review] HVDetFusion: A Simple and Robust Camera-Radar Fusion Framework

Kai Lei, Zhan Chen+2 more|arXiv (Cornell University)|Jul 21, 2023

Advanced Neural Network Applications4 citations

TL;DR

HVDetFusion is a novel, decoupled camera-radar fusion framework for 3D object detection in autonomous driving, leveraging a BevDet4D-based backbone for camera-only inference and fusing radar data via spatial-temporal priors to reduce false positives. It achieves state-of-the-art 67.4% NDS on the nuScenes test set, outperforming existing camera-radar detectors, especially on small objects like pedestrians and traffic cones.

ABSTRACT

In the field of autonomous driving, 3D object detection is a very important perception module. Although the current SOTA algorithm combines Camera and Lidar sensors, limited by the high price of Lidar, the current mainstream landing schemes are pure Camera sensors or Camera+Radar sensors. In this study, we propose a new detection algorithm called HVDetFusion, which is a multi-modal detection algorithm that not only supports pure camera data as input for detection, but also can perform fusion input of radar data and camera data. The camera stream does not depend on the input of Radar data, thus addressing the downside of previous methods. In the pure camera stream, we modify the framework of Bevdet4D for better perception and more efficient inference, and this stream has the whole 3D detection output. Further, to incorporate the benefits of Radar signals, we use the prior information of different object positions to filter the false positive information of the original radar data, according to the positioning information and radial velocity information recorded by the radar sensors to supplement and fuse the BEV features generated by the original camera data, and the effect is further improved in the process of fusion training. Finally, HVDetFusion achieves the new state-of-the-art 67.4\% NDS on the challenging nuScenes test set among all camera-radar 3D object detectors. The code is available at https://github.com/HVXLab/HVDetFusion

Motivation & Objective

• To develop a flexible, robust, and efficient multi-modal 3D object detection framework that supports both pure camera and camera-radar fusion inference.
• To address the limitation of prior fusion methods that require radar input for camera stream operation, enabling standalone camera performance.
• To improve detection accuracy—especially for small and challenging objects—by effectively fusing radar data using spatial and radial velocity priors.
• To achieve state-of-the-art performance on the challenging nuScenes benchmark using only camera and radar sensors, avoiding expensive LiDAR.

Proposed method

• Adopts a two-stream, decoupled architecture: a BevDet4D-based stream for pure camera inference and a separate radar fusion stream.
• Uses spatial and radial velocity priors from radar to filter false positives in raw radar detections before fusion.
• Fuses radar-enhanced features with camera-generated BEV features through a learnable cross-attention mechanism during training.
• Employs a multi-frame temporal modeling strategy with 8 future frames to improve motion and velocity estimation.
• Utilizes advanced backbones (ConvNeXt-Base, Internimage-Base) and high-resolution input (640×1600) to boost feature representation.
• Applies COCO pre-training and ablation-driven hyperparameter tuning to optimize training efficiency and performance.

Experimental results

Research questions

• RQ1 Can a decoupled camera-radar fusion framework maintain strong standalone camera performance while enabling effective radar fusion?
• RQ2 How does incorporating radar priors (position and radial velocity) improve BEV feature quality and reduce false positives?
• RQ3 What is the impact of temporal modeling (multi-frame input) and backbone choice on 3D detection accuracy in camera-radar fusion?
• RQ4 To what extent does high-resolution input and pre-training improve detection performance on small objects like pedestrians and barriers?

Key findings

• HVDetFusion achieves a new state-of-the-art 67.4% NDS on the nuScenes test set among all camera-radar 3D object detectors.
• The model improves detection by 4.7% on Trucks, 5.8% on Buses, and 5.6% on Motorcycles compared to BevDet-Gamma.
• It achieves 8.6% higher mAP on Pedestrians, 7.8% on Traffic Cones, and 12.1% on Barriers compared to CRN, demonstrating strong gains on small objects.
• Fusion with radar boosts NDS from 55.7% (camera-only with 8 future frames) to 65.2%, proving the effectiveness of the radar fusion module.
• Using ConvNeXt-Base or Internimage-Base backbones increases NDS to 58.7%, showing the benefit of stronger feature extractors.
• Higher input resolution (640×1600) improves NDS by 5.4% compared to 256×704, highlighting the importance of spatial detail.