[Paper Review] MeshDiffusion: Score-based Generative 3D Mesh Modeling

Zhen Liu, Feng Yao+4 more|arXiv (Cornell University)|Mar 14, 2023

3D Shape Modeling and Analysis29 citations

TL;DR

MeshDiffusion trains a diffusion model directly on deformable tetrahedral grids to generate high-quality 3D meshes, enabling unconditional and conditional mesh generation with a simple, stable pipeline.

ABSTRACT

We consider the task of generating realistic 3D shapes, which is useful for a variety of applications such as automatic scene generation and physical simulation. Compared to other 3D representations like voxels and point clouds, meshes are more desirable in practice, because (1) they enable easy and arbitrary manipulation of shapes for relighting and simulation, and (2) they can fully leverage the power of modern graphics pipelines which are mostly optimized for meshes. Previous scalable methods for generating meshes typically rely on sub-optimal post-processing, and they tend to produce overly-smooth or noisy surfaces without fine-grained geometric details. To overcome these shortcomings, we take advantage of the graph structure of meshes and use a simple yet very effective generative modeling method to generate 3D meshes. Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parametrization. We demonstrate the effectiveness of our model on multiple generative tasks.

Motivation & Objective

• Demonstrate that diffusion models can directly generate high-quality 3D meshes represented on deformable tetrahedral grids.
• Provide a simple, stable training and inference pipeline that leverages a 3D-CNN U-Net on a uniform tetrahedral grid.
• Enable unconditional generation, conditional generation from single-view RGBD, and interpolation of 3D meshes.
• Show that per-vertex SDF-based conditioning with normalization stabilizes diffusion-based mesh generation.

Proposed method

• Represent meshes with deformable tetrahedral grids (DMTet) and model vertex deformations and SDF values directly.
• Use a 3D U-Net diffusion model trained on the tetrahedral grid with augmented cubic-grid structure to exploit spatial priors.
• Normalize SDF values to ±1 and train a diffusion model on continuous deformation while applying a two-pass optimization to stabilize mesh surfaces.
• Render-guided objective L_Render (image, depth, Chamfer, penalties) to connect mesh generation with differentiable rendering.
• During inference, solve the reverse SDE to generate meshes and optionally refine SDF signs and perform a conditional generation pass.
• Enable conditional generation from a single RGBD view via a two-stage process: fit a tetrahedral grid to the view and complete with the diffusion model.

Experimental results

Research questions

• RQ1 Can diffusion models be effectively applied to generate high-quality 3D meshes directly on a deformable tetrahedral grid?
• RQ2 Does normalizing SDF values and using a two-pass training scheme stabilize mesh generation and preserve fine geometric details?
• RQ3 How well does MeshDiffusion perform in unconditional generation, conditional generation from single views, and interpolation tasks?
• RQ4 What is the impact of the proposed DMTet-based parametrization and 3D CNN architecture on training stability and surface detail fidelity?

Key findings

• MeshDiffusion yields sharp, detailed meshes and preserves fine geometry better than SDF-based baselines in qualitative comparisons.
• Quantitative metrics (e.g., point-cloud distances and light-field distance) show MeshDiffusion achieving competitive or superior results to baselines across ShapeNet categories.
• A normalization strategy for SDFs and a simple, non-amortized two-stage training approach improve stability and reduce artifacts from marching tetrahedra.
• Conditional generation from a single RGBD view demonstrates plausible shape completion while allowing minor vertex updates during diffusion.
• Ablation studies indicate the SDF normalization and diffusion-on-grid design contribute meaningfully to performance.