[Paper Review] PointCNN: Convolution On $\mathcal{X}$-Transformed Points

Yangyan Li, Rui Bu+4 more|arXiv (Cornell University)|Jan 23, 2018

3D Shape Modeling and Analysis60 references123 citations

TL;DR

PointCNN introduces an 𝒳-Conv operator that learns a point-wise 𝒳-transformation to weight and permute neighborhood features, enabling CNN-like learning on irregular point clouds and achieving competitive results across classification and segmentation benchmarks.

ABSTRACT

We present a simple and general framework for feature learning from point clouds. The key to the success of CNNs is the convolution operator that is capable of leveraging spatially-local correlation in data represented densely in grids (e.g. images). However, point clouds are irregular and unordered, thus directly convolving kernels against features associated with the points, will result in desertion of shape information and variance to point ordering. To address these problems, we propose to learn an $\mathcal{X}$-transformation from the input points, to simultaneously promote two causes. The first is the weighting of the input features associated with the points, and the second is the permutation of the points into a latent and potentially canonical order. Element-wise product and sum operations of the typical convolution operator are subsequently applied on the $\mathcal{X}$-transformed features. The proposed method is a generalization of typical CNNs to feature learning from point clouds, thus we call it PointCNN. Experiments show that PointCNN achieves on par or better performance than state-of-the-art methods on multiple challenging benchmark datasets and tasks.

Motivation & Objective

• Promote learning of spatially-local correlations in point clouds without imposing a regular grid structure.
• Develop an 𝒳-transformation to weight and permute neighborhood features prior to convolution.
• Generalize CNNs to point clouds and build hierarchical architectures for classification and segmentation.

Proposed method

• Introduce 𝒳-Conv, which computes a learned 𝑘×𝑘 𝒳-transformation from local neighborhood coordinates.
• Lift point coordinates with an auxiliary MLP to form 𝒳-Conv input features, then apply a learned 𝑘×𝑘 𝒳-transformation to weight and permute features before standard convolution.
• Construct hierarchical PointCNN architectures by alternating 𝒳-Conv layers with point pooling/downsampling and optional dilations for larger receptive fields.
• Use local coordinate systems centered at representative points to ensure the output depends on relative geometry rather than absolute position.
• Optionally fuse global coordinates via an additional MLP before subsequent 𝒳-Conv layers.
• Train end-to-end with standard backpropagation and data augmentation via random sampling/shuffling of neighborhood points.

Experimental results

Research questions

• RQ1 Can an 𝒳-transformation learn to weight and permute neighborhood features to recover shape information lost by naive pointwise convolution?
• RQ2 Does PointCNN generalize CNN-like feature learning to irregular, unordered point clouds while remaining differentiable and trainable end-to-end?
• RQ3 What architectural choices (density of connections, dilation, skip connections) improve performance on classification and segmentation tasks?
• RQ4 How does PointCNN compare to other point-cloud networks and image-based CNNs across standard benchmarks?

Key findings

• PointCNN achieves competitive to state-of-the-art results on ModelNet40 and ScanNet for classification with 1024 input points.
• PointCNN outperforms several methods in ShapeNet Parts segmentation across pIoU, mpIoU, and OA metrics.
• Ablation studies show 𝒳-Conv is the key contributor to performance over variants without 𝒳-transformations.
• Visualization suggests 𝒳-transformations concentrate features and promote discriminative representations for representative points.
• Dense/dual-path versions with dilations enable broader receptive fields while maintaining feasible parameter counts.
• PointCNN demonstrates strong performance on sketch datasets and maintains reasonable results on MNIST/CIFAR10 when using point-cloud representations.
• Compared with PointNet/PointNet++ and other 3D architectures, PointCNN offers favorable accuracy with a moderate parameter and FLOP footprint.