[Paper Review] OccFusion: Multi-Sensor Fusion Framework for 3D Semantic Occupancy Prediction

Zhenxing Ming, Julie Stephany Berrío+2 more|arXiv (Cornell University)|Mar 3, 2024

Video Surveillance and Tracking Methods4 citations

TL;DR

OccFusion proposes a dynamic multi-sensor fusion framework integrating 360° camera, lidar, and radar data for 3D semantic occupancy prediction, using novel 3D/2D dynamic fusion modules to enhance robustness and accuracy. It achieves state-of-the-art performance on nuScenes, especially in challenging night and rainy conditions, outperforming camera-only baselines by up to 6.5% mIoU gain with full sensor fusion.

ABSTRACT

A comprehensive understanding of 3D scenes is crucial in autonomous vehicles (AVs), and recent models for 3D semantic occupancy prediction have successfully addressed the challenge of describing real-world objects with varied shapes and classes. However, existing methods for 3D occupancy prediction heavily rely on surround-view camera images, making them susceptible to changes in lighting and weather conditions. This paper introduces OccFusion, a novel sensor fusion framework for predicting 3D occupancy. By integrating features from additional sensors, such as lidar and surround view radars, our framework enhances the accuracy and robustness of occupancy prediction, resulting in top-tier performance on the nuScenes benchmark. Furthermore, extensive experiments conducted on the nuScenes and semanticKITTI dataset, including challenging night and rainy scenarios, confirm the superior performance of our sensor fusion strategy across various perception ranges. The code for this framework will be made available at https://github.com/DanielMing123/OccFusion.

Motivation & Objective

• To address the limitations of camera-only 3D occupancy prediction under adverse lighting and weather conditions.
• To integrate complementary strengths of lidar (geometric accuracy) and radar (weather robustness) with camera (semantic richness) for improved scene understanding.
• To develop a unified, efficient fusion framework that enhances model robustness across diverse perception ranges and environmental conditions.
• To evaluate the performance gains of multi-sensor fusion strategies in challenging scenarios such as night and rain using the nuScenes benchmark.

Proposed method

• Proposes a dynamic fusion 3D/2D module that combines BEV features from cameras with 3D feature volumes from lidar and radar using learnable attention mechanisms.
• Employs a multi-scale coarse-to-fine refinement structure with multi-level supervision to improve feature representation at all spatial levels.
• Introduces a BEV feature as a global excitation signal to guide 3D feature fusion and accelerate convergence.
• Uses SENet blocks in both 2D and 3D spaces to dynamically recalibrate multi-modal features, enhancing critical feature amplification.
• Supports three fusion strategies: Camera+Radar, Camera+Lidar, and Camera+Lidar+Radar, enabling ablation and comparative analysis.
• Applies feature-level fusion in bird’s-eye view (BEV) space before lifting to 3D, enabling effective cross-modality interaction.

Figure 1 : The figure above demonstrates the pipeline of two approaches: The purely vision-centric approach (top) and the multi-sensor fusion approach (bottom). We conduct 3D semantic occupancy prediction by doing feature fusion with respect to three modality feature volumes.

Experimental results

Research questions

• RQ1 How does fusing lidar and radar with camera data improve 3D semantic occupancy prediction under adverse lighting and weather conditions?
• RQ2 What is the relative contribution of each sensor modality (camera, lidar, radar) to the final prediction performance across different perception ranges?
• RQ3 How does the proposed dynamic fusion 3D/2D module compare to simple concatenation or fixed attention mechanisms in multi-modal feature fusion?
• RQ4 What is the impact of multi-scale supervision and refinement on model convergence and performance?
• RQ5 How does the full sensor fusion strategy (camera + lidar + radar) perform in extreme scenarios like night and rain compared to single- or dual-modality baselines?

Key findings

• The full OccFusion framework (Camera+Lidar+Radar) achieves a mean Intersection over Union (mIoU) of 30.97% on the nuScenes validation set, outperforming camera-only baselines.
• In night and rainy scenarios, the addition of lidar and radar improves mIoU by up to 6.5 percentage points compared to camera-only models.
• The multi-scale coarse-to-fine refinement structure with multi-level supervision increases mIoU by 5.23 percentage points compared to a single-scale baseline.
• Removing the BEV feature in the dynamic fusion module causes a 6.5% drop in mIoU, demonstrating its critical role in guiding 3D feature fusion.
• The SENet3D and SENet2D blocks significantly boost performance, with removal reducing mIoU by over 6% when both are disabled.
• The framework maintains strong performance across different perception ranges, with the Camera+Lidar+Radar fusion strategy showing consistent gains at all ranges.

Figure 2 : Overall architecture of OccFusion. Firstly, the surround-view images were inputted into the 2D backbone to extract multiple-scale features. Subsequently, a view transformation is conducted at each scale to obtain the global BEV feature and the local 3D feature volume at each level. The 3D