[Paper Review] PointConv: Deep Convolutional Networks on 3D Point Clouds

Wenxuan Wu, Zhongang Qi+1 more|arXiv (Cornell University)|Nov 17, 2018

3D Shape Modeling and Analysis47 references103 citations

TL;DR

PointConv introduces a density-reweighted, MLP-driven convolution for 3D point clouds, achieving translation- and permutation-invariant 3D CNNs with a memory-efficient implementation and a deconvolution counterpart for segmentation.

ABSTRACT

Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and density functions through kernel density estimation. The most important contribution of this work is a novel reformulation proposed for efficiently computing the weight functions, which allowed us to dramatically scale up the network and significantly improve its performance. The learned convolution kernel can be used to compute translation-invariant and permutation-invariant convolution on any point set in the 3D space. Besides, PointConv can also be used as deconvolution operators to propagate features from a subsampled point cloud back to its original resolution. Experiments on ModelNet40, ShapeNet, and ScanNet show that deep convolutional neural networks built on PointConv are able to achieve state-of-the-art on challenging semantic segmentation benchmarks on 3D point clouds. Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure.

Motivation & Objective

• Motivate applying convolution to irregular, unordered 3D point clouds without converting to grids.
• Propose PointConv to approximate continuous 3D convolutions via learned weight and density functions.
• Enable scalable, permutation-invariant and translation-invariant convolution on point sets.
• Extend PointConv to a deconvolution operator for propagating features in segmentation tasks.
• Demonstrate state-of-the-art or strong performance on synthetic and real 3D datasets and compare to 2D CNN benchmarks on CIFAR-10 converted to point clouds.

Proposed method

• Treat convolution as a continuous operator and approximate the weight function with an MLP over local coordinates.
• Estimate inverse density via kernel density estimation and transform it with an MLP to reweight contributions (Monte Carlo rationale).
• Share weights across points to preserve permutation invariance and apply a memory-efficient reformulation that reduces PointConv to matrix multiplication and 1x1 convolutions.
• Compute PointConv over local neighborhoods using a KNN-based grouping and a 1x1 MLP for weight generation; aggregate via weighted sums with the density scale.
• Introduce PointDeconv to propagate coarse features to finer resolutions through interpolation followed by PointConv-based refinement.
• Demonstrate scalability and performance across ModelNet40, ShapeNet Part, ScanNet, and CIFAR-10 paradigms (point clouds and converted images).

Experimental results

Research questions

• RQ1 Can a continuous, density-weighted convolution defined on 3D points achieve translation- and permutation-invariance comparable to grid-based CNNs?
• RQ2 How can PointConv be implemented efficiently to scale to modern network sizes and resolutions?
• RQ3 Does a deconvolution counterpart (PointDeconv) improve segmentation by leveraging coarse-to-fine feature propagation?
• RQ4 How does PointConv perform on challenging 3D semantic segmentation benchmarks and on CIFAR-10 when treated as a point-cloud problem?
• RQ5 What is the impact of inverse density scaling and MLP configuration on performance and memory usage?

Key findings

• PointConv achieves state-of-the-art-like performance on ShapeNet Part and competitive results on ModelNet40 when using 3D point inputs.
• On ScanNet semantic scene labeling, PointConv outperforms several baselines with a mean IoU of 55.6% (vs. 30.6%–43.8% for baselines).
• PointConv can match 2D CNN performance on CIFAR-10 when using a 5-layer or VGG-like structure applied to point clouds.
• The efficient PointConv formulation reduces memory from an intractable 8 GB per layer to a practical level, enabling deeper networks.
• Inverse density scaling improves results by about 1% in ablations, particularly in earlier layers; the nonlinear transform is crucial for effective density handling.
• PointDeconv enables effective upsampling/propagation of features for segmentation tasks, leveraging skip connections and coarse-to-fine information.