[Paper Review] PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Charles R. Qi, Yi Li+2 more|arXiv (Cornell University)|Jun 7, 2017

Image Processing and 3D Reconstruction7,051 citations

TL;DR

PointNet++ introduces a hierarchical network that processes point sets in a metric space by recursively applying PointNet on nested local regions, enabling multi-scale, density-adaptive feature learning for 3D point clouds and non-Euclidean data.

ABSTRACT

Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.

Motivation & Objective

• Motivate learning on point sets that reside in a metric space and are often non-uniformly sampled.
• Develop a hierarchical architecture to capture local structures at multiple scales.
• Address non-uniform sampling density with adaptive, multi-scale feature aggregation.
• Leverage PointNet as a local feature learner within a hierarchical framework.
• Demonstrate robustness and efficiency across 2D/3D datasets and non-Euclidean spaces.

Proposed method

• Define a hierarchical neural network that partitions the point set into overlapping local regions using a metric-space distance.
• Use farthest point sampling (FPS) to select centroids for receptive fields.
• Group neighboring points with ball queries and encode local regions with a mini-PointNet.
• Build multiple set abstraction levels that progressively abstract to higher-level features.
• Introduce multi-scale grouping (MSG) and multi-resolution grouping (MRG) to handle varying densities.
• Incorporate random input dropout during training to learn adaptive weighting across scales, and propagate features back to original points via distance-based interpolation with skip connections.

Experimental results

Research questions

• RQ1 Can a hierarchical, metric-space-aware architecture capture local geometric structures at multiple scales for point sets?
• RQ2 How can non-uniform sampling densities be handled without sacrificing fine-grained detail?
• RQ3 Do density-adaptive multi-scale strategies improve robustness and accuracy on benchmarks with real-world, dense-to-sparse point clouds?
• RQ4 Is PointNet++ effective for both Euclidean and non-Euclidean (intrinsic metric) point sets?
• RQ5 What is the impact of multi-scale and density-aware design on semantic segmentation and object classification tasks?

Key findings

• The hierarchical architecture significantly improves over vanilla PointNet on MNIST and ModelNet40 classifications.
• Multi-scale and density-adaptive strategies enable robust performance across varying sampling densities.
• MSG and DP (random input dropout during training) yield strong robustness to density variation, with minimal performance loss when test density decreases.
• MRG offers a computationally efficient alternative to MSG while preserving adaptive information aggregation across densities.
• PointNet++ achieves state-of-the-art or competitive results on 3D point cloud benchmarks including ScanNet for semantic scene labeling and SHREC15 for non-rigid shape classification.
• The approach generalizes to non-Euclidean metric spaces by using intrinsic features and geodesic neighborhoods to capture intrinsic structure.