[Paper Review] From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network

Shaoshuai Shi, Zhe Wang+3 more|arXiv (Cornell University)|Jul 8, 2019

Advanced Neural Network Applications63 references99 citations

TL;DR

Introduces Part-A2 net, a two-stage 3D object detector for LiDAR point clouds that leverages intra-object part locations and RoI-aware pooling to improve 3D detection, achieving state-of-the-art results on KITTI using only point cloud data.

ABSTRACT

3D object detection from LiDAR point cloud is a challenging problem in 3D scene understanding and has many practical applications. In this paper, we extend our preliminary work PointRCNN to a novel and strong point-cloud-based 3D object detection framework, the part-aware and aggregation neural network (Part-$A^2$ net). The whole framework consists of the part-aware stage and the part-aggregation stage. Firstly, the part-aware stage for the first time fully utilizes free-of-charge part supervisions derived from 3D ground-truth boxes to simultaneously predict high quality 3D proposals and accurate intra-object part locations. The predicted intra-object part locations within the same proposal are grouped by our new-designed RoI-aware point cloud pooling module, which results in an effective representation to encode the geometry-specific features of each 3D proposal. Then the part-aggregation stage learns to re-score the box and refine the box location by exploring the spatial relationship of the pooled intra-object part locations. Extensive experiments are conducted to demonstrate the performance improvements from each component of our proposed framework. Our Part-$A^2$ net outperforms all existing 3D detection methods and achieves new state-of-the-art on KITTI 3D object detection dataset by utilizing only the LiDAR point cloud data. Code is available at https://github.com/sshaoshuai/PointCloudDet3D.

Motivation & Objective

• Leverage free intra-object part location supervision derived from 3D ground-truth boxes to learn discriminative 3D point features.
• Develop a two-stage detector that uses part information to propose and refine 3D boxes from point clouds.
• Introduce RoI-aware point cloud pooling to preserve geometry-specific information for accurate box refinement.

Proposed method

• Two-stage framework with a part-aware stage (Stage-I) and a part-aggregation stage (Stage-II).
• Stage-I learns foreground segmentation and intra-object part locations, and generates 3D proposals via anchor-free or anchor-based strategies.
• Stage-II uses RoI-aware pooling to aggregate part features and employs sparse convolutions for box scoring and refinement.
• Intra-object part locations are defined as relative 3D positions of foreground points within ground-truth boxes and are learned with corresponding losses.
• Anchor-free proposal generation uses a bin-based center regression and residual refinements to predict object centers and orientation.
• Anchor-based proposal generation uses a Region Proposal Network on bird-view features with predefined 3D anchors and a residual-based regression loss.

Experimental results

Research questions

• RQ1 How can free intra-object part location information from 3D bounding boxes enhance 3D object detection from point clouds?
• RQ2 Can a two-stage detector with RoI-aware pooling outperform single-stage and other two-stage methods using only LiDAR data?
• RQ3 What are effective strategies for 3D proposal generation (anchor-free vs anchor-based) in point-cloud-based detection?
• RQ4 Does a differentiable RoI-aware pooling operation improve box scoring and localization refinement?
• RQ5 How does the Part-A2 net perform on standard benchmarks like KITTI compared to existing methods?

Key findings

• Part-A2 net achieves state-of-the-art 3D detection performance on KITTI using only LiDAR point cloud data.
• The part-aware stage simultaneously predicts intra-object part locations and generates 3D proposals.
• The part-aggregation stage with RoI-aware pooling improves proposal scoring and localization refinement via learned part features.
• Two proposal-generation strategies (anchor-free and anchor-based) address different deployment needs; anchor-free is memory-efficient while anchor-based offers higher recall.
• The framework runs at 14 FPS on KITTI as of August 15, 2019, outperforming published methods at the time.