[Paper Review] 3DGen: Triplane Latent Diffusion for Textured Mesh Generation

Anchit Gupta, Wenhan Xiong+3 more|arXiv (Cornell University)|Mar 9, 2023

3D Shape Modeling and Analysis33 citations

TL;DR

3DGen introduces a two-stage pipeline with a triplane VAE and a diffusion model to generate high-quality textured meshes across categories, enabling image-, text-, and unconditional generation with strong scalability.

ABSTRACT

Latent diffusion models for image generation have crossed a quality threshold which enabled them to achieve mass adoption. Recently, a series of works have made advancements towards replicating this success in the 3D domain, introducing techniques such as point cloud VAE, triplane representation, neural implicit surfaces and differentiable rendering based training. We take another step along this direction, combining these developments in a two-step pipeline consisting of 1) a triplane VAE which can learn latent representations of textured meshes and 2) a conditional diffusion model which generates the triplane features. For the first time this architecture allows conditional and unconditional generation of high quality textured or untextured 3D meshes across multiple diverse categories in a few seconds on a single GPU. It outperforms previous work substantially on image-conditioned and unconditional generation on mesh quality as well as texture generation. Furthermore, we demonstrate the scalability of our model to large datasets for increased quality and diversity. We will release our code and trained models.

Motivation & Objective

• Motivate scalable, high-quality 3D textured mesh generation beyond single-category models.
• Develop a two-stage architecture that learns compact triplane latent representations of textured meshes and generates them via diffusion.
• Enable both texture-enabled and texture-less mesh generation, conditioned on images or text, with efficient GPU-backed inference.

Proposed method

• Train a triplane VAE that encodes colored point clouds into 3D-aware triplane latent features and decodes to textured meshes via differentiable marching tetrahedra.
• Train a diffusion model operating on rolled-out triplanes to generate triplane features conditioned on image-text embeddings or unconditionally.
• Use rendering-based losses with a differentiable renderer and DMTet for training the VAE, plus KL, smoothing, and reconstruction terms.
• Condition diffusion on image embeddings via a frozen image-text bi-encoder and AdaGN-based conditioning without cross-attention, enabling scalable text-conditioned generation.
• Finetune diffusion with higher-resolution tetrahedral grid during later stages to improve fine details and shading.

Experimental results

Research questions

• RQ1 Can a two-stage framework combining a triplane VAE and a diffusion model learn high-quality, conditional and unconditional textured meshes across diverse categories?
• RQ2 Does conditioning diffusion on image or text embeddings improve geometry and texture fidelity compared to unconditional generation?
• RQ3 How does data scale (e.g., pre-training on large Objaverse-like datasets) affect geometry accuracy and texture quality?
• RQ4 What architectural choices (conditioning method, VAE resolution, surface texture modeling) most impact mesh quality and texture fidelity?

Key findings

• 3DGen yields substantial improvements over prior work in multiple settings: unconditional geometry generation shows 23% FiD improvement over NFD, unconditional colored mesh generation up to 70% FiD improvement, and text-conditioned geometry generation improvements of 15-20% over SoTA 3DILG.
• The two-stage triplane approach enables conditional and unconditional textured or untextured mesh generation across categories in seconds on a single GPU.
• Pre-training on large datasets (Objaverse + 3D Warehouse) improves Chamfer distances and FiD scores, with more pronounced gains for low-resource categories.
• Text-conditioned generation using text embeddings demonstrates well-formed object generation aligned with prompts, showcasing the method’s flexibility with simple prompt-based conditioning.