[Paper Review] PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation

Danfei Xu, Dragomir Anguelov+1 more|arXiv (Cornell University)|Nov 29, 2017

3D Surveying and Cultural Heritage32 references55 citations

TL;DR

PointFusion combines image and raw 3D point cloud data with a heterogeneous network (PointNet + CNN) and a dense fusion predictor to estimate 3D bounding boxes, achieving competitive results across KITTI and SUN-RGBD without dataset-specific tuning.

ABSTRACT

We present PointFusion, a generic 3D object detection method that leverages both image and 3D point cloud information. Unlike existing methods that either use multi-stage pipelines or hold sensor and dataset-specific assumptions, PointFusion is conceptually simple and application-agnostic. The image data and the raw point cloud data are independently processed by a CNN and a PointNet architecture, respectively. The resulting outputs are then combined by a novel fusion network, which predicts multiple 3D box hypotheses and their confidences, using the input 3D points as spatial anchors. We evaluate PointFusion on two distinctive datasets: the KITTI dataset that features driving scenes captured with a lidar-camera setup, and the SUN-RGBD dataset that captures indoor environments with RGB-D cameras. Our model is the first one that is able to perform better or on-par with the state-of-the-art on these diverse datasets without any dataset-specific model tuning.

Motivation & Objective

• Motivate a generic 3D object detection approach that works across diverse sensors and environments without dataset-specific tuning.
• Propose an architecture that fuses image features with raw point cloud data using heterogeneous networks.
• Introduce a dense fusion mechanism using spatial anchors to predict 3D box corners and select the best hypothesis.

Proposed method

• Process image crops with a CNN to extract appearance/geometry features.
• Process raw 3D point clouds with a PointNet variant to obtain point-wise and global features.
• Fuse image and point features via a fusion network to predict 3D bounding boxes.
• Use two fusion variants: a vanilla global regressor of the eight box corners and a dense fusion that predicts corner offsets relative to input 3D points.
• Apply a dense spatial anchor approach where each input point yields a corner-offset prediction with a learned score, selecting the best prediction.
• Train with a regression loss on corner locations plus a spatial transform regularizer; explore supervised and unsupervised scoring for point selection.

Experimental results

Research questions

• RQ1 Can a generic, sensor-agnostic fusion model achieve competitive 3D object detection across outdoor (KITTI) and indoor (SUN-RGBD) datasets?
• RQ2 Does the dense fusion architecture with spatial anchors outperform a traditional global corner regression in 3D box estimation?
• RQ3 How does fusing image data with raw point clouds compare to using either modality alone on diverse object categories (cars, pedestrians, cyclists, etc.)?
• RQ4 What is the impact of supervised versus unsupervised scoring for selecting the best dense prediction?

Key findings

• PointFusion with dense fusion and image+point inputs consistently outperforms lidar-only baselines on KITTI and SUN-RGBD for multiple classes.
• Dense fusion using spatial anchors (points as anchors) substantially improves 3D box regression over direct corner regression.
• Unsupervised scoring for selecting the best hypothesis often yields better results than supervised scoring in several settings.
• The method achieves competitive AP3D on KITTI (cars, pedestrians, cyclists) and SUN-RGBD across 10–19 categories, often surpassing or matching state-of-the-art methods while being computationally efficient.
• Fusing RGB data with LiDAR-like point clouds yields the most significant gains for smaller objects (pedestrians, cyclists) where depth points are sparse.