[Paper Review] Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

Tianchang Shen, Jun Gao+3 more|arXiv (Cornell University)|Nov 8, 2021

3D Shape Modeling and Analysis66 references105 citations

TL;DR

DMTet is a deep 3D conditional generator that upscales coarse voxels into high-resolution meshes using a deformable tetrahedral grid and a differentiable Marching Tetrahedra layer, achieving topology changes and greater detail than prior implicit or explicit methods.

ABSTRACT

We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.

Motivation & Objective

• Motivate democratizing high-quality 3D content creation by upscaling coarse voxel shapes (e.g., Minecraft) into detailed 3D assets.
• Introduce a hybrid implicit-explicit 3D representation that supports arbitrary topology and high resolution.
• Enable end-to-end training with surface-focused supervision to improve geometric detail and realism.

Proposed method

• Represent shapes with a sign distance field defined on a deformable tetrahedral grid.
• Use a differentiable Marching Tetrahedra layer to convert the implicit SDF to an explicit surface mesh.
• Grow and adapt the tetrahedral grid via volume subdivision around the predicted surface to support high resolution.
• Refine the surface with a learnable Loop-like subdivision in a surface-parameterized mesh.
• Train with a combination of surface alignment (Chamfer and normal) loss, adversarial loss, and regularizations on SDF and deformations.
• Discriminator operates on SDFs derived from predicted meshes to assess realism.

Experimental results

Research questions

• RQ1 Can DMTet reconstruct and synthesize high-resolution 3D shapes with arbitrary topology from coarse inputs?
• RQ2 Does a surface-supervised, differentiable MT-based pipeline improve geometric detail and realism over purely implicit or explicit methods?
• RQ3 How does volume and surface subdivision, plus learnable surface refinement, affect reconstruction quality and topology flexibility?
• RQ4 Is end-to-end training of a hybrid implicit-explicit surface representation more efficient and scalable than prior approaches for high-resolution 3D synthesis?

Key findings

• DMTet significantly outperforms ConvOnet and DECOR baselines on high-resolution animal-shape synthesis from coarse voxels across multiple metrics.
• Including adversarial loss improves realism and fine geometric details.
• Volume subdivision helps capture fine-grained structures (e.g., airplane, lamp); surface subdivision reduces quantization and yields smoother surfaces.
• DMTet achieves competitive/inferior-time performance, being faster than many implicit-method baselines at inference.
• Ablations show deforming the grid and joint SDF+surface supervision are crucial for high-quality results.