[Paper Review] PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection

Shaoshuai Shi, Li Jiang+6 more|arXiv (Cornell University)|Jan 31, 2021

Advanced Neural Network Applications89 references28 citations

TL;DR

PV-RCNN++ advances 3D object detection by tightly coupling point-based and voxel-based features, introducing sectorized sampling and VectorPool aggregation for faster and more accurate detection on LiDAR data.

ABSTRACT

3D object detection is receiving increasing attention from both industry and academia thanks to its wide applications in various fields. In this paper, we propose Point-Voxel Region-based Convolution Neural Networks (PV-RCNNs) for 3D object detection on point clouds. First, we propose a novel 3D detector, PV-RCNN, which boosts the 3D detection performance by deeply integrating the feature learning of both point-based set abstraction and voxel-based sparse convolution through two novel steps, i.e., the voxel-to-keypoint scene encoding and the keypoint-to-grid RoI feature abstraction. Second, we propose an advanced framework, PV-RCNN++, for more efficient and accurate 3D object detection. It consists of two major improvements: sectorized proposal-centric sampling for efficiently producing more representative keypoints, and VectorPool aggregation for better aggregating local point features with much less resource consumption. With these two strategies, our PV-RCNN++ is about $3 imes$ faster than PV-RCNN, while also achieving better performance. The experiments demonstrate that our proposed PV-RCNN++ framework achieves state-of-the-art 3D detection performance on the large-scale and highly-competitive Waymo Open Dataset with 10 FPS inference speed on the detection range of 150m * 150m.

Motivation & Objective

• Motivate improved 3D object detection on point clouds by combining point- and voxel-based feature learning.
• Develop a two-stage detector (PV-RCNN) that deeply fuses voxel-to-keypoint encoding with keypoint-to-grid RoI features.
• Introduce PV-RCNN++ with sectorized sampling and VectorPool aggregation for efficiency and accuracy on large-scale scenes.

Proposed method

• Use a 3D sparse convolution backbone to generate multi-scale voxel features for initial proposals.
• Summarize voxel features at keypoints via Voxel Set Abstraction to form multi-scale keypoint features.
• Aggregate keypoint features to RoI-grid points through RoI-grid pooling for fine-grained refinement.
• Apply Predicted Keypoint Weighting to emphasize foreground keypoints during refinement (optional).
• PV-RCNN++ adds Sectorized Proposal-Centric sampling to speed up keypoint sampling and VectorPool aggregation for efficient local feature encoding.

Experimental results

Research questions

• RQ1 Can deep fusion of voxel-based and point-based features improve 3D detection accuracy over voxel- or point-based methods alone?
• RQ2 Do sectorized keypoint sampling and VectorPool aggregation reduce inference time and memory usage while maintaining or improving performance on large-scale datasets?
• RQ3 How does the proposed RoI-grid pooling strategy affect proposal refinement quality compared to previous RoI methods?
• RQ4 What is the impact of foreground-focused keypoint weighting on detection results?

Key findings

• PV-RCNN++ is about 3x faster than PV-RCNN while achieving better performance.
• It achieves state-of-the-art results on the Waymo Open Dataset in a 150m x 150m range.
• The framework enables 10 FPS inference for large-range detection.
• Sectorized sampling concentrates keypoints around proposals and accelerates sampling.
• VectorPool aggregation provides memory-efficient, position-sensitive local feature encoding for large point clouds.