[Paper Review] PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows

Guandao Yang, Xun Huang+4 more|arXiv (Cornell University)|Jun 28, 2019

3D Shape Modeling and AnalysisEngineering56 references72 citations

TL;DR

PointFlow models 3D point clouds as a distribution of distributions using two stacked continuous normalizing flows to represent shape distribution and point distribution given a shape, enabling end-to-end likelihood-based training and high-fidelity generation.

ABSTRACT

As 3D point clouds become the representation of choice for multiple vision and graphics applications, the ability to synthesize or reconstruct high-resolution, high-fidelity point clouds becomes crucial. Despite the recent success of deep learning models in discriminative tasks of point clouds, generating point clouds remains challenging. This paper proposes a principled probabilistic framework to generate 3D point clouds by modeling them as a distribution of distributions. Specifically, we learn a two-level hierarchy of distributions where the first level is the distribution of shapes and the second level is the distribution of points given a shape. This formulation allows us to both sample shapes and sample an arbitrary number of points from a shape. Our generative model, named PointFlow, learns each level of the distribution with a continuous normalizing flow. The invertibility of normalizing flows enables the computation of the likelihood during training and allows us to train our model in the variational inference framework. Empirically, we demonstrate that PointFlow achieves state-of-the-art performance in point cloud generation. We additionally show that our model can faithfully reconstruct point clouds and learn useful representations in an unsupervised manner. The code will be available at https://github.com/stevenygd/PointFlow.

Motivation & Objective

• Learn a probabilistic model for 3D point clouds as a distribution of distributions (shapes and points within shapes).
• Enable sampling of arbitrary-point-count point clouds from learned shapes.
• Train end-to-end using variational inference with invertible flows for exact likelihood computation.
• Provide a principled alternative to GAN-based methods for point cloud generation.

Proposed method

• Model the shape distribution with a continuous normalizing flow (CNF) over a latent shape variable z.
• Model the point distribution given a shape with a conditional CNF G_theta that maps a prior y( t0 ) ~ N(0,I) to points x conditioned on z.
• Use a CNF-based prior over shapes z via a learnable CNF F_psi operating on a Gaussian prior w.
• Train with a variational lower bound (ELBO) that includes the prior term, reconstruction term, and posterior entropy.
• Sample by drawing z through F_psi and then generating points via G_theta conditioned on z, starting from a Gaussian prior.
• Provide exact log-likelihoods through CNF density estimation and enable end-to-end optimization.

Experimental results

Research questions

• RQ1 Can a distribution-of-distributions framework (shapes and points per shape) improve point cloud generation quality and flexibility?
• RQ2 Does using continuous normalizing flows for both shape prior and point generation enable principled likelihood-based training and better reconstructions?
• RQ3 How does PointFlow compare to GAN-based point cloud generators on standard metrics and unsupervised representation learning?

Key findings

• PointFlow achieves state-of-the-art performance on point cloud generation across ShapeNet categories and outperforms baselines on 1-NNA and other metrics.
• The model enables sampling of arbitrary numbers of points from a learned shape, addressing fixed-size point set limitations.
• The CNF-based prior over shapes and the conditional CNF for point generation allow closed-form likelihoods, enabling stable end-to-end training without GANs.
• The approach yields competitive unsupervised learned representations for downstream tasks (e.g., ModelNet classification).
• PointFlow uses substantially fewer parameters than some baselines while maintaining or improving performance.