[Paper Review] Deep Continuous Fusion for Multi-Sensor 3D Object Detection

Ming Liang, Bin Yang+2 more|arXiv (Cornell University)|Dec 20, 2020

Advanced Neural Network Applications39 references430 citations

TL;DR

Introduces a two-stream, end-to-end 3D object detector that continuously fuses camera image features into a LIDAR BEV backbone using a continuous fusion layer for improved multi-sensor 3D localization.

ABSTRACT

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.

Motivation & Objective

• Motivate robust 3D object detection using complementary camera and LIDAR data in autonomous driving.
• Develop a learnable fusion mechanism that preserves geometric information across modalities.
• Enable end-to-end training with continuous, multi-scale fusion for BEV-based detection.
• Demonstrate real-time performance and strong accuracy on KITTI and TOR4D benchmarks.

Proposed method

• Propose a dual-stream network with image and LIDAR BEV branches.
• Introduce a continuous fusion layer that projects image features into BEV and fuses them with LIDAR BEV features via a KNN-based interpolation and an MLP that incorporates 3D offsets.
• Use deep parametric continuous convolution to interpolate dense BEV features from sparse image-LIDAR correspondences.
• Fuse multi-scale image features into BEV across four fusion layers in a feature pyramid-style BEV backbone.
• Train end-to-end with a multi-task loss combining classification and regression terms for 3D bounding boxes and orientation.

Experimental results

Research questions

• RQ1 Can continuous fusion of image and LIDAR features in BEV space improve 3D object detection over LIDAR-only and coarse fusion baselines?
• RQ2 How do KNN pooling and geometric offset features affect cross-modal fusion performance?
• RQ3 What are the trade-offs between accuracy and real-time inference with multi-scale continuous fusion?

Key findings

• Outperforms state-of-the-art methods on KITTI BEV and competitive on 3D detection, with strong real-time performance (>15 FPS).
• KITTI results show Our Cont Fuse achieves 3D AP easy 82.54, moderate 66.22, hard 64.04 and BEV AP easy 88.81, moderate 85.83, hard 77.33.
• TOR4D results show strong long-range performance with multi-class BEV detection (Vehicle AP0.5 94.94, Vehicle AP0.7 75.34; Pedestrian AP0.3 83.89, AP0.5 74.08; Bicyclist AP0.3 82.32, AP0.5 59.83).
• Compared to LIDAR-only and discrete fusion baselines, continuous fusion with KNN pooling and geometric offsets yields consistent gains across metrics.
• Ablation studies show both KNN pooling and the geometric offset input are important; removing either degrades performance.
• Long-range advantages are pronounced on TOR4D, particularly when x increases, indicating effective fusion for distant objects.