[Paper Review] LION: Latent Point Diffusion Models for 3D Shape Generation

Xiaohui Zeng, Arash Vahdat+5 more|arXiv (Cornell University)|Oct 12, 2022

3D Shape Modeling and AnalysisEngineering189 citations

TL;DR

LION introduces a hierarchical latent diffusion framework for 3D shape generation that uses a two-stage VAE with latent point and global shape latents, achieving state-of-the-art results on ShapeNet benchmarks and enabling flexible tasks like multimodal denoising, voxel-conditioned synthesis, and mesh reconstruction.

ABSTRACT

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space. For generation, we train two hierarchical DDMs in these latent spaces. The hierarchical VAE approach boosts performance compared to DDMs that operate on point clouds directly, while the point-structured latents are still ideally suited for DDM-based modeling. Experimentally, LION achieves state-of-the-art generation performance on multiple ShapeNet benchmarks. Furthermore, our VAE framework allows us to easily use LION for different relevant tasks: LION excels at multimodal shape denoising and voxel-conditioned synthesis, and it can be adapted for text- and image-driven 3D generation. We also demonstrate shape autoencoding and latent shape interpolation, and we augment LION with modern surface reconstruction techniques to generate smooth 3D meshes. We hope that LION provides a powerful tool for artists working with 3D shapes due to its high-quality generation, flexibility, and surface reconstruction. Project page and code: https://nv-tlabs.github.io/LION.

Motivation & Objective

• Aim to develop a high-quality, flexible 3D shape generator suitable for artists and designers.
• Leverage a hierarchical VAE with latent diffusion models to model global shape and point-level details.
• Enable multimodal and conditional 3D generation (e.g., voxel-guided, text/image-driven) without retraining the diffusion models.
• Provide mesh-ready outputs by integrating with modern surface reconstruction techniques.
• Demonstrate state-of-the-art performance on ShapeNet benchmarks and scalability to multi-class, multi-category data.

Proposed method

• Model 3D shapes as point clouds x ∈ R^{3×N} using a hierarchical VAE with a global latent z0 and a latent point cloud h0 ∈ R^{(3+Dh)×N}.
• Train two latent diffusion models in the latent spaces: one for z0 (global shape) and one conditional on z0 for h0 (latent points).
• First stage: train encoders/decoders with ELBO-based objectives and Gaussian priors; second stage: freeze VAE and train latent DDMs via score-matching objectives (L_SM^z, L_SM^h).
• Use Point-Voxel CNNs (PVCNNs) for encoders/decoders and a mixed score parametrization to model latent distributions toward Gaussian priors.
• Enable diffusion-denoise in latent space to control detail levels (diffuse-denoise) and combine with SAP-based surface reconstruction to produce smooth meshes.
• Allow fine-tuning of encoders for voxel-conditioned synthesis and multimodal denoising without re-training latent DDMs, and support shape interpolation via probability-flow ODEs.

Experimental results

Research questions

• RQ1 Can a hierarchical VAE with latent diffusion in both a global shape and a latent point space achieve state-of-the-art 3D shape synthesis?
• RQ2 Does latent-space diffusion provide better expressivity and flexibility for multimodal, voxel-guided, and mesh-enabled generation compared to point-cloud DDMs?
• RQ3 How well does LION scale to multi-class and highly multimodal ShapeNet data without conditioning?
• RQ4 Can LION be effectively integrated with surface reconstruction to output smooth meshes suitable for artists?
• RQ5 To what extent can latent diffusion enable controlled variation and denoising tasks (multimodal denoise, interpolation) without re-training the diffusion models?

Key findings

• LION achieves state-of-the-art generation performance on ShapeNet benchmarks, outperforming several baselines (PVD, DPM) on multiple metrics.
• A two-stage training (VAE plus latent DDMs) and hierarchical latent spaces yield high expressivity and better sampling quality than diffusion on raw point clouds.
• Unconditional multi-class LION models (13 and 55 classes) generate diverse, plausible shapes without conditioning, demonstrating strong mode coverage.
• Coupling LION with Shape As Points (SAP) enables smooth meshing, and diffuse-denoise in latent space allows controlled variation of details while preserving shape.
• Voxel-guided synthesis and multimodal denoising are feasible by fine-tuning encoders while keeping latent DDMs fixed, enabling interactive editing and refinement.
• Tables and qualitative results show LION’s superiority over baselines in CD/EMD metrics across airplanes, chairs, and cars (Table 3: ShapeNet-vol).
• Mesh outputs produced via SAP appear smooth and semantically coherent, illustrating practical utility for artists.