[Paper Review] Cross-Modality 3D Object Detection

Ming Zhu, Chao Ma+2 more|arXiv (Cornell University)|Aug 16, 2020

Robotics and Sensor-Based Localization31 references4 citations

TL;DR

This paper proposes a two-stage, multi-modal 3D object detection framework that fuses binocular images and raw LiDAR point clouds to improve detection accuracy. By employing point-wise feature fusion, a 2D-3D coupling loss, and pseudo-LiDAR point augmentation, the method enhances sparse point cloud representations and achieves state-of-the-art performance on the KITTI dataset, particularly for distant and occluded objects.

ABSTRACT

In this paper, we focus on exploring the fusion of images and point clouds for 3D object detection in view of the complementary nature of the two modalities, i.e., images possess more semantic information while point clouds specialize in distance sensing. To this end, we present a novel two-stage multi-modal fusion network for 3D object detection, taking both binocular images and raw point clouds as input. The whole architecture facilitates two-stage fusion. The first stage aims at producing 3D proposals through sparse point-wise feature fusion. Within the first stage, we further exploit a joint anchor mechanism that enables the network to utilize 2D-3D classification and regression simultaneously for better proposal generation. The second stage works on the 2D and 3D proposal regions and fuses their dense features. In addition, we propose to use pseudo LiDAR points from stereo matching as a data augmentation method to densify the LiDAR points, as we observe that objects missed by the detection network mostly have too few points especially for far-away objects. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.

Motivation & Objective

• Address the challenge of sparse and unevenly distributed LiDAR point clouds in 3D object detection.
• Leverage the complementary strengths of images (rich semantics) and LiDAR (precise depth) for improved 3D localization.
• Overcome limitations of single-modality detection by fusing multi-modal features at multiple stages.
• Improve detection performance for distant and occluded objects, which suffer from point sparsity.
• Develop a data augmentation strategy using pseudo-LiDAR points to densify sparse ground truth point clouds.

Proposed method

• Utilizes PointNet++ to directly learn 3D representations from raw LiDAR point clouds without voxelization or projection.
• Employs a modified ResNet-50 with FPN as the image backbone to extract discriminative 2D feature maps from binocular images.
• Introduces a joint anchor mechanism that enables simultaneous 2D and 3D classification and regression for improved 3D proposal generation.
• Applies a 2D-3D reprojection loss to couple 2D detection constraints with 3D proposals, improving proposal quality through cross-modality supervision.
• Performs RoI-wise feature fusion to combine dense 2D and 3D features within proposal regions for robust representation learning.
• Generates pseudo-LiDAR point clouds from stereo matching and uses them as data augmentation by adaptively cropping and adding to real point clouds.

Experimental results

Research questions

• RQ1 Can a two-stage fusion framework effectively combine 2D image semantics and 3D LiDAR geometry for improved 3D object detection?
• RQ2 How does a 2D-3D coupling loss that enforces consistency between 2D and 3D predictions affect proposal quality and final detection accuracy?
• RQ3 To what extent can pseudo-LiDAR points generated from binocular images mitigate the sparsity problem in distant object detection?
• RQ4 Does RoI-wise fusion of multi-modal features lead to more robust and accurate 3D bounding box predictions compared to early or late fusion?
• RQ5 Can the proposed method outperform existing single-modality or single-stage fusion approaches on standard benchmarks like KITTI?

Key findings

• The proposed method achieves state-of-the-art performance on the KITTI 3D object detection benchmark, with a 3D AP of 86.80% on the moderate set for cars.
• The 2D-3D coupling loss reduces the number of mispredicted objects (IoU=0) by aligning 3D proposals with 2D detections, improving localization consistency.
• Proposal recall improves significantly across all difficulty levels after applying the reprojection loss, especially for hard examples.
• Fusion with pseudo-LiDAR points boosts performance on hard and moderate classes, with a 2.09% AP gain on the hard car set when using GT-cropped pseudo points.
• The method shows particular strength in detecting distant and occluded objects due to enhanced point cloud density from pseudo-LiDAR augmentation.
• The ablation study confirms that both the joint anchor mechanism and pseudo-LiDAR augmentation contribute significantly to performance gains, with the latter being most effective for sparse regions.