[Paper Review] BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

Tingting Liang, Hongwei Xie+7 more|arXiv (Cornell University)|May 27, 2022

Advanced Neural Network ApplicationsComputer Science201 citations

TL;DR

BEVFusion separates camera and LiDAR streams into a shared BEV space and fuses them with a dynamic module, improving robustness to LiDAR or camera malfunctions and achieving state-of-the-art results on nuScenes without post-processing.

ABSTRACT

Fusing the camera and LiDAR information has become a de-facto standard for 3D object detection tasks. Current methods rely on point clouds from the LiDAR sensor as queries to leverage the feature from the image space. However, people discovered that this underlying assumption makes the current fusion framework infeasible to produce any prediction when there is a LiDAR malfunction, regardless of minor or major. This fundamentally limits the deployment capability to realistic autonomous driving scenarios. In contrast, we propose a surprisingly simple yet novel fusion framework, dubbed BEVFusion, whose camera stream does not depend on the input of LiDAR data, thus addressing the downside of previous methods. We empirically show that our framework surpasses the state-of-the-art methods under the normal training settings. Under the robustness training settings that simulate various LiDAR malfunctions, our framework significantly surpasses the state-of-the-art methods by 15.7% to 28.9% mAP. To the best of our knowledge, we are the first to handle realistic LiDAR malfunction and can be deployed to realistic scenarios without any post-processing procedure. The code is available at https://github.com/ADLab-AutoDrive/BEVFusion.

Motivation & Objective

• Identify limitations of existing LiDAR-camera fusion methods that rely on LiDAR inputs for image queries.
• Propose a two-stream framework that encodes camera and LiDAR data into a common BEV representation independent of each other.
• Develop a simple, effective fusion module to combine BEV features from both modalities.
• Demonstrate robustness to realistic LiDAR malfunctions and camera failures through specialized augmentation and evaluation.

Proposed method

• Two independent streams encode inputs into the same BEV space: a camera stream (based on Lift-Splat-Shoot adaptations with a Dual-Swin-Tiny backbone and a BEV encoder) and a LiDAR stream (using PointPillars, CenterPoint, or TransFusion as backbones).
• A dynamic fusion module concatenates BEV features and applies a channel-spatial fusion followed by adaptive feature selection to produce F_fused.
• The fused BEV feature feeds standard 3D detection heads (anchor-based, anchor-free, or transformer-based) for predictions.
• Camera BEV features are generated via a view projector that predicts depth and renders image features into a voxel grid, then compressed by an BEV encoder using a Spatial-to-Channel approach.
• The LiDAR BEV features are produced by established LiDAR backbones, converted to BEV without dependence on the camera branch.
• Training uses a two-stage process: train the monocular and LiDAR streams separately, then fine-tune the BEVFusion with both streams.

Experimental results

Research questions

• RQ1 Can a LiDAR-camera fusion framework be designed that does not rely on LiDAR inputs to generate camera queries and still achieve strong performance?
• RQ2 Does disentangling camera and LiDAR streams improve robustness to LiDAR malfunctions and camera failures compared with traditional fusion methods?
• RQ3 How well does BEVFusion generalize across different LiDAR backbones and detection heads?
• RQ4 What is the impact of the proposed dynamic fusion module on detection accuracy across modalities?

Key findings

• BEVFusion improves several LiDAR-based detectors when fused with camera streams, e.g., PointPillars gains 18.4% mAP and CenterPoint gains 3.0%–7.1% mAP depending on the baseline.
• On nuScenes test, BEVFusion with TransFusion-L as LiDAR backbone achieves 69.2% mAP, surpassing TransFusion’s 68.9% and the state-of-the-art without test-time augmentation.
• Under LiDAR robustness experiments (limited FOV and object LiDAR-point dropping), BEVFusion consistently outperforms LiDAR-only baselines and vanilla LC fusion, with gains up to 25+ mAP in some settings.
• Under camera robustness scenarios, BEVFusion remains competitive and often superior to camera-only and other fusion baselines when cameras are missing or degraded.
• Ablation studies show the camera stream benefits from a larger 2D backbone (Dual-Swin-Tiny) and that the dynamic fusion module (CSF + AFS) significantly boosts performance over static fusion.