[Paper Review] MeshSDF: Differentiable Iso-Surface Extraction

Edoardo Remelli, Artem Lukoianov+5 more|arXiv (Cornell University)|Jun 6, 2020

3D Shape Modeling and Analysis55 references22 citations

TL;DR

MeshSDF introduces a differentiable iso-surface extraction method that enables end-to-end backpropagation through mesh generation from deep signed distance functions (SDFs), overcoming the non-differentiability of traditional algorithms like Marching Cubes. By deriving a closed-form gradient of surface samples w.r.t. the implicit field, MeshSDF enables topology-varying, high-resolution mesh optimization in tasks like single-view reconstruction and aerodynamic shape design, outperforming state-of-the-art methods in both accuracy and efficiency.

ABSTRACT

Geometric Deep Learning has recently made striking progress with the advent of continuous Deep Implicit Fields. They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D Euclidean grid, resulting in a learnable parameterization that is not limited in resolution. Unfortunately, these methods are often not suitable for applications that require an explicit mesh-based surface representation because converting an implicit field to such a representation relies on the Marching Cubes algorithm, which cannot be differentiated with respect to the underlying implicit field. In this work, we remove this limitation and introduce a differentiable way to produce explicit surface mesh representations from Deep Signed Distance Functions. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples with respect to the underlying deep implicit field. We exploit this to define MeshSDF, an end-to-end differentiable mesh representation which can vary its topology. We use two different applications to validate our theoretical insight: Single-View Reconstruction via Differentiable Rendering and Physically-Driven Shape Optimization. In both cases our differentiable parameterization gives us an edge over state-of-the-art algorithms.

Motivation & Objective

• To address the non-differentiability of Marching Cubes in converting deep implicit fields to explicit mesh representations.
• To enable end-to-end differentiable optimization over explicit surface meshes while preserving topological flexibility and high resolution.
• To demonstrate the utility of differentiable mesh parameterization in applications requiring explicit geometry, such as differentiable rendering and CFD-based shape optimization.
• To provide a theoretically grounded, closed-form gradient for surface samples w.r.t. the underlying implicit field, independent of the extraction algorithm.
• To enable robust and efficient optimization in complex physical simulation and vision tasks by combining the expressiveness of SDFs with the compatibility of mesh-based representations.

Proposed method

• Derives a closed-form analytical expression for the gradient of 3D surface samples w.r.t. the underlying deep SDF, enabling backpropagation through the mesh extraction process.
• Uses a non-differentiable iso-surface extraction method (e.g., Marching Cubes) to generate the mesh, then applies a custom backward pass using the derived gradient formula.
• Employs a differentiable rasterization pipeline to enable differentiable rendering and silhouette-based optimization in single-view reconstruction.
• Introduces a latent space regularization loss based on nearest-neighbor latent codes to stabilize optimization and improve generalization in shape space.
• Applies the MeshSDF parameterization as a regularizer in physics-driven optimization, such as minimizing pressure drag in CFD simulations.
• Utilizes the Adam optimizer for optimization, with hyperparameters tuned for convergence and robustness across diverse tasks.

Experimental results

Research questions

• RQ1 Can we differentiate through the iso-surface extraction process of deep SDFs to produce differentiable mesh representations?
• RQ2 Does a closed-form gradient of surface samples w.r.t. the implicit field enable end-to-end optimization over meshes with arbitrary topology?
• RQ3 Can MeshSDF outperform non-differentiable mesh-based methods in single-view 3D reconstruction with differentiable rendering?
• RQ4 Can MeshSDF serve as an effective regularizer in physics-based shape optimization, such as aerodynamic drag minimization?
• RQ5 How does the differentiability of MeshSDF compare to alternative differentiable rendering or implicit field methods in terms of efficiency and accuracy?

Key findings

• MeshSDF achieves a silhouette L2 distance of 0.004625 in single-view reconstruction, outperforming Liu et al. (2020) with 0.005973 under comparable settings.
• The method reduces network queries from 898k (sphere tracing) to 266k, achieving a 76% reduction in function evaluations.
• MeshSDF runs in 0.29 seconds per render, a 77% speedup compared to Liu et al.'s 1.24 seconds, due to parallel rasterization of triangles.
• In CFD-based aerodynamic optimization, MeshSDF successfully refines car shapes to minimize pressure drag, demonstrating utility in physics-driven design.
• The method enables topology changes during optimization, unlike fixed-topology alternatives, and shows robustness across different initial shapes.
• Latent space regularization with k=10 and α=0.2 improves optimization stability and realism, though local minima in latent space remain a challenge.