[Paper Review] Diffusion Probabilistic Models for 3D Point Cloud Generation

Shitong Luo, Wei Hu|arXiv (Cornell University)|Mar 2, 2021

3D Shape Modeling and Analysis26 references24 citations

TL;DR

A diffusion-based probabilistic model generates 3D point clouds by learning a reverse Markov diffusion process conditioned on a shape latent, achieving competitive generation and auto-encoding performance. It uses a variational lower bound objective and a latent prior via normalizing flows.

ABSTRACT

We present a probabilistic model for point cloud generation, which is fundamental for various 3D vision tasks such as shape completion, upsampling, synthesis and data augmentation. Inspired by the diffusion process in non-equilibrium thermodynamics, we view points in point clouds as particles in a thermodynamic system in contact with a heat bath, which diffuse from the original distribution to a noise distribution. Point cloud generation thus amounts to learning the reverse diffusion process that transforms the noise distribution to the distribution of a desired shape. Specifically, we propose to model the reverse diffusion process for point clouds as a Markov chain conditioned on certain shape latent. We derive the variational bound in closed form for training and provide implementations of the model. Experimental results demonstrate that our model achieves competitive performance in point cloud generation and auto-encoding. The code is available at \url{https://github.com/luost26/diffusion-point-cloud}.

Motivation & Objective

• Motivate robust generative modeling for irregular 3D point clouds.
• Propose a diffusion-based reverse process conditioned on a shape latent to generate point clouds.
• Derive a tractable variational objective for training the conditional diffusion model.
• Demonstrate competitive performance in point cloud generation and auto-encoding.
• Showcase representation learning capabilities for unsupervised tasks.

Proposed method

• Model forward diffusion as a Markov chain that adds Gaussian noise to points.
• Model reverse diffusion as a trainable Gaussian Markov kernel conditioned on a shape latent z.
• Parameterize the shape latent prior p(z) with normalizing flows for flexibility.
• Derive a closed-form variational lower bound and tractable training objective.
• Use PointNet for the encoder q(z|X) and affine coupling flows to transform z distributions.
• Apply the framework to point cloud generation, auto-encoding, and unsupervised representation learning.

Experimental results

Research questions

• RQ1 Can a diffusion-based reverse process conditioned on a shape latent accurately generate diverse 3D point clouds?
• RQ2 Does conditioning on a latent z improve generation and enable flexible priors via normalizing flows?
• RQ3 Is the variational lower bound tractable and effective for training diffusion models on point clouds?
• RQ4 How does the proposed method perform on generation, auto-encoding, and representation learning tasks compared to state-of-the-art?
• RQ5 What is the impact of the latent prior and encoder design on reconstruction and latent representations?

Key findings

• The model achieves competitive performance in point cloud generation and auto-encoding.
• The diffusion-based reverse process can generate meaningful 3D shapes when conditioned on a shape latent.
• Normalizing flows provide a flexible prior for the latent shape variable, improving expressiveness.
• The training objective is tractable and derived from a variational lower bound.
• The encoder and diffusion framework yield informative latent representations for unsupervised tasks.
• Latent space interpolations and visualizations demonstrate meaningful structure in the learned representations.