[Paper Review] PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection

Shaoshuai Shi, Chaoxu Guo+5 more|arXiv (Cornell University)|Dec 31, 2019

3D Shape Modeling and Analysis46 references184 citations

TL;DR

PV-RCNN integrates voxel-based and PointNet-based features to achieve state-of-the-art 3D object detection from LiDAR point clouds, generating high-quality proposals and refined RoI features.

ABSTRACT

We present a novel and high-performance 3D object detection framework, named PointVoxel-RCNN (PV-RCNN), for accurate 3D object detection from point clouds. Our proposed method deeply integrates both 3D voxel Convolutional Neural Network (CNN) and PointNet-based set abstraction to learn more discriminative point cloud features. It takes advantages of efficient learning and high-quality proposals of the 3D voxel CNN and the flexible receptive fields of the PointNet-based networks. Specifically, the proposed framework summarizes the 3D scene with a 3D voxel CNN into a small set of keypoints via a novel voxel set abstraction module to save follow-up computations and also to encode representative scene features. Given the high-quality 3D proposals generated by the voxel CNN, the RoI-grid pooling is proposed to abstract proposal-specific features from the keypoints to the RoI-grid points via keypoint set abstraction with multiple receptive fields. Compared with conventional pooling operations, the RoI-grid feature points encode much richer context information for accurately estimating object confidences and locations. Extensive experiments on both the KITTI dataset and the Waymo Open dataset show that our proposed PV-RCNN surpasses state-of-the-art 3D detection methods with remarkable margins by using only point clouds. Code is available at https://github.com/open-mmlab/OpenPCDet.

Motivation & Objective

• Motivate high-accuracy 3D object detection from sparse, irregular LiDAR point clouds.
• Propose a unified framework that combines 3D voxel CNNs with PointNet-style set abstraction.
• Develop voxel-to-keypoint scene encoding and keypoint-to-grid RoI feature abstraction for efficient, rich representations.
• Enable accurate proposal refinement and confidence estimation through multi-scale context aggregation.
• Demonstrate state-of-the-art performance on KITTI and Waymo Open datasets using only point clouds.

Proposed method

• Use a 3D voxel CNN with sparse convolutions as backbone for efficient feature encoding and 3D proposal generation.
• Summarize voxel CNN features over the whole scene into a small set of keypoints via voxel set abstraction (VSA).
• Aggregate multi-scale voxel features to each keypoint using multi-radius neighborhood sets and PointNet-style pooling.
• Enrich keypoint features with raw point and BEV (bird-eye view) features and optionally apply Predicted Keypoint Weighting (PKW) for foreground emphasis.
• Perform RoI-grid pooling: for each 3D proposal, sample a 6x6x6 grid and use set abstraction with multi-radius neighborhoods to fuse keypoint features into RoI-grid features.
• Train end-to-end with losses L_rpn (proposal generation), L_seg (keypoint segmentation), and L_rcnn (proposal refinement including IoU-guided confidence).

Experimental results

Research questions

• RQ1 Can a hybrid point-voxel architecture outperform pure voxel- or pure point-based detectors on LiDAR-only 3D detection?
• RQ2 Does summarizing voxel CNN features into a small set of keypoints enable efficient yet rich feature aggregation for RoI refinement?
• RQ3 Can multi-scale, radius-based set abstraction improve proposal refinement and confidence estimation compared with traditional RoI pooling?
• RQ4 What is the impact of enriching keypoints with raw points and BEV features, and of Weighted keypoints (PKW) on detection performance?
• RQ5 How does PV-RCNN perform on large-scale datasets (KITTI, Waymo Open) compared to prior state-of-the-art?]
• RQ6 key_findings([

Key findings

• PV-RCNN achieves state-of-the-art performance on KITTI car 3D detection, ranking 1st among LiDAR-only methods as of the reported date.
• On KITTI, PV-RCNN improves 3D mAP by 1.58/1.72/1.73 points over prior methods for Easy/Moderate/Hard car detection respectively.
• PV-RCNN also attains top BEV mAP on Easy and Moderate car detection, with a slight drop on Hard, and surpasses previous LiDAR-only methods for cyclist detection in Moderate/Hard.
• Table 1 shows PV-RCNN achieving 3D mAP of 90.25 (Easy), 81.43 (Moderate), 76.82 (Hard) for cars, BEV mAP of 94.98 (Easy), 90.65 (Moderate), 86.14 (Hard), and cyclist metrics of 78.60 (Easy), 63.71 (Moderate), 57.65 (Hard) 3D mAP, with BEV 82.49 (Easy), 68.89 (Moderate), 62.41 (Hard).
• PV-RCNN outperforms prior methods on the KITTI val split (R11 and R40) and demonstrates strong performance on Waymo Open Test/Validation sets (details in the paper).