[Paper Review] Learning Deformable Tetrahedral Meshes for 3D Reconstruction

Jun Gao, Wenzheng Chen+5 more|arXiv (Cornell University)|Nov 3, 2020

3D Shape Modeling and AnalysisEngineering55 references49 citations

TL;DR

DefTet introduces a deformable tetrahedral mesh representation for 3D reconstruction, enabling joint optimization of vertex positions and tetrahedron occupancy to produce high-fidelity, topologically flexible tetrahedral meshes directly from point clouds or single images. It achieves comparable or better quality with lower memory and faster inference than prior volumetric and mesh-based methods.

ABSTRACT

3D shape representations that accommodate learning-based 3D reconstruction are an open problem in machine learning and computer graphics. Previous work on neural 3D reconstruction demonstrated benefits, but also limitations, of point cloud, voxel, surface mesh, and implicit function representations. We introduce Deformable Tetrahedral Meshes (DefTet) as a particular parameterization that utilizes volumetric tetrahedral meshes for the reconstruction problem. Unlike existing volumetric approaches, DefTet optimizes for both vertex placement and occupancy, and is differentiable with respect to standard 3D reconstruction loss functions. It is thus simultaneously high-precision, volumetric, and amenable to learning-based neural architectures. We show that it can represent arbitrary, complex topology, is both memory and computationally efficient, and can produce high-fidelity reconstructions with a significantly smaller grid size than alternative volumetric approaches. The predicted surfaces are also inherently defined as tetrahedral meshes, thus do not require post-processing. We demonstrate that DefTet matches or exceeds both the quality of the previous best approaches and the performance of the fastest ones. Our approach obtains high-quality tetrahedral meshes computed directly from noisy point clouds, and is the first to showcase high-quality 3D tet-mesh results using only a single image as input. Our project webpage: https://nv-tlabs.github.io/DefTet/

Motivation & Objective

• Motivate learning-based 3D reconstruction with a representation that supports arbitrary topology and internal volume.
• Propose Deformable Tetrahedral Meshes (DefTet) that jointly optimize vertex positions and tetrahedron occupancy in a differentiable framework.
• Demonstrate DefTet’s effectiveness on point clouds, single images, tetrahedral meshing benchmarks, and novel view synthesis.
• Show that DefTet can produce tetrahedral meshes directly (no post-processing) and scale memory/computation more efficiently than traditional voxel-based volumes.

Proposed method

• Define a fixed initial tetrahedralization of the unit cube using QuarTet.
• Predict per-vertex offsets and per-tetrahedron occupancy to deform the mesh and classify inside/outside regions.
• Compute surface probability for each face from neighboring tetrahedra occupancies (Eq. 2).
• Train with differentiable losses that couple occupancy, deformation, and surface alignment (recon3D, recon2D, laplacian, equi-volume, AMIPS, etc.).
• Use a differentiable renderer to project DefTet to 2D when only images are available (soft z-buffer-like visibility and color via barycentric interpolation).
• Optionally perform 3D or 2D supervision (occupancy BCE for 3D; L1 image loss for 2D).
• Infer by thresholding tetrahedron occupancies and extracting the surface mesh from shared faces.

Experimental results

Research questions

• RQ1 Can a deformable, occupancy-aware tetrahedral mesh represent objects with arbitrary topology and high geometric detail more efficiently than voxel grids or fixed-topology meshes?
• RQ2 Is it possible to learn end-to-end a DefTet representation directly from noisy point clouds or single images without heavy post-processing?
• RQ3 How does DefTet perform across 3D supervision, 2D supervision, and differentiable rendering settings for reconstruction quality and speed?
• RQ4 What are the benefits of joint deformation and occupancy optimization for downstream tasks like meshing, novel view synthesis, and multi-view reconstruction?

Key findings

• DefTet can represent arbitrary topology and yields high-fidelity reconstructions with a smaller grid size than equivalent volumetric approaches.
• DefTet produces surfaces as tetrahedral meshes directly, avoiding post-processing during inference.
• DefTet matches or exceeds quality of prior best approaches while offering faster inference, e.g., being significantly faster than Occupancy Networks.
• DefTet demonstrates high-quality tetrahedral meshes from noisy point clouds and achieves firsts in single-image 3D reconstruction using tetrahedral meshes.
• In multi-view and novel-view contexts, DefTet converges faster than Nerf-based methods while providing usable geometries for downstream tasks (e.g., physics-based simulation).
• Across point-cloud and image inputs, DefTet shows competitive 3D IoU and Chamfer-based metrics with substantially lower memory footprint than high-resolution voxel grids.