[Paper Review] Bi-LRFusion: Bi-Directional LiDAR-Radar Fusion for 3D Dynamic Object Detection

Yingjie Wang, Jiajun Deng+7 more|arXiv (Cornell University)|Jun 2, 2023

Advanced SAR Imaging Techniques4 citations

TL;DR

Bi-LRFusion proposes a bi-directional LiDAR-Radar fusion framework for 3D dynamic object detection that addresses the lack of height information in radar data by learning pseudo-height features from LiDAR points via a query-based height feature fusion (QHF) block and enhances radar BEV features using query-based BEV feature fusion (QBF). The method achieves state-of-the-art performance, with a 2.3% AP gain on similar-height objects and 1.3% on tall objects, and outperforms LiDAR-only baselines at long ranges beyond 40m.

ABSTRACT

LiDAR and Radar are two complementary sensing approaches in that LiDAR specializes in capturing an object's 3D shape while Radar provides longer detection ranges as well as velocity hints. Though seemingly natural, how to efficiently combine them for improved feature representation is still unclear. The main challenge arises from that Radar data are extremely sparse and lack height information. Therefore, directly integrating Radar features into LiDAR-centric detection networks is not optimal. In this work, we introduce a bi-directional LiDAR-Radar fusion framework, termed Bi-LRFusion, to tackle the challenges and improve 3D detection for dynamic objects. Technically, Bi-LRFusion involves two steps: first, it enriches Radar's local features by learning important details from the LiDAR branch to alleviate the problems caused by the absence of height information and extreme sparsity; second, it combines LiDAR features with the enhanced Radar features in a unified bird's-eye-view representation. We conduct extensive experiments on nuScenes and ORR datasets, and show that our Bi-LRFusion achieves state-of-the-art performance for detecting dynamic objects. Notably, Radar data in these two datasets have different formats, which demonstrates the generalizability of our method. Codes are available at https://github.com/JessieW0806/BiLRFusion.

Motivation & Objective

• Address the challenge of radar's inability to capture object height, which leads to inconsistent detection performance across object types.
• Improve 3D object detection accuracy for both similar-height and tall objects by fusing LiDDR and radar features in a bidirectional manner.
• Enhance detection performance at long ranges where LiDAR point clouds become sparse, leveraging radar's consistent long-range coverage.
• Develop a robust fusion mechanism tolerant to sensor misalignment, ensuring reliable performance under real-world calibration errors.
• Demonstrate generalizability by applying the framework to multiple LiDAR-only detectors, including CenterPoint and TransFusion-L.

Proposed method

• Introduces a query-based height feature fusion (QHF) block that learns pseudo-height features from LiDAR points to compensate for radar's lack of height information.
• Employs a query-based BEV feature fusion (QBF) block to refine radar BEV features using detailed LiDAR features, improving detection of small or distant objects.
• Uses a bi-directional fusion strategy: R2L (Radar-to-LiDAR) and L2R (LiDAR-to-Radar) to enable cross-modal feature enhancement.
• Applies ball query and Manhattan distance-based BEV queries to aggregate features within spatial neighborhoods, ensuring localized and context-aware fusion.
• Leverages a PointNet and MLP-based backbone for feature extraction from both LiDAR and radar point clouds, with output channels set to 32.
• Employs a multi-scale fusion mechanism that dynamically selects relevant features based on query locations, improving robustness to sensor misalignment.

Experimental results

Research questions

• RQ1 How does the absence of height information in radar data affect 3D object detection performance across objects of varying heights?
• RQ2 To what extent can learning pseudo-height features from LiDAR improve radar-based detection of tall objects?
• RQ3 Can query-based fusion mechanisms enhance radar BEV features using detailed LiDAR features, especially for small or distant objects?
• RQ4 How does the proposed fusion framework perform at long ranges where LiDAR point density decreases?
• RQ5 How tolerant is the fusion framework to sensor misalignment in real-world autonomous driving scenarios?

Key findings

• The QHF block improves mean AP by 2.3% on similar-height objects and 1.3% on tall objects compared to a LiDAR-only baseline, demonstrating its effectiveness in compensating for radar’s height limitation.
• The QBF block alone increases AP by 4.1% on similar-height objects and 0.4% on tall objects, showing that detailed LiDAR features enhance radar BEV representation.
• At ranges beyond 40m, Bi-LRFusion achieves a 2.2% higher mAP on cars compared to the LiDAR-only CenterPoint baseline, highlighting its advantage in long-range detection.
• The framework maintains strong performance under sensor misalignment up to level 3, with only a 0.6% drop in mAP, and degrades significantly at level 4, indicating a practical tolerance limit.
• When applied to TransFusion-L, Bi-LRFusion improves mAP from 65.3% to 67.5% and reduces mAVE from 24.8 to 23.7, confirming its generalizability across different detection architectures.