[Paper Review] NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

Peng Wang, Lingjie Liu+4 more|arXiv (Cornell University)|Jun 20, 2021

3D Shape Modeling and Analysis544 citations

TL;DR

NeuS learns neural signed distance functions (SDFs) and trains them with a novel unbiased volume rendering scheme, enabling high-fidelity multi-view surface reconstruction without foreground masks.

ABSTRACT

We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.

Motivation & Objective

• Motivate robust surface reconstruction from multi-view images beyond mask-supervised, surface-rendering approaches.
• Represent surfaces as zero-level sets of neural SDFs and train via a new volume rendering formulation.
• Ensure unbiased first-order surface reconstruction while preserving occlusion handling.
• Demonstrate strong reconstruction quality on challenging datasets (DTU, BlendedMVS) with and without masks.

Proposed method

• Represent scene with a neural SDF f and a color field c parameterized by MLPs.
• Introduce S-density phi_s derived from a logistic distribution to connect SDF to volume rendering.
• Define an occlusion-aware, unbiased weight w(t) along rays using an opaque density rho and standard volume rendering framework.
• Provide equations to compute rho(t) and w(t) (rho(t)=max(-dPhi_s/dt / Phi_s, 0); w(t)=T(t) rho(t) with T(t)=exp(-∫ rho).
• Train end-to-end with color loss plus optional mask loss and an Eikonal regularization on the SDF; use hierarchical sampling as in NeRF.
• Handle background with NeRF++ when w/o mask setting; initialize geometry and regularize with Eikonal and geometric priors.

Experimental results

Research questions

• RQ1 Can an SDF-based representation be effectively trained with volume rendering to produce high-quality surfaces from 2D images without mask supervision?
• RQ2 Does a bias-free, occlusion-aware volume rendering formulation improve surface reconstruction fidelity over conventional density-based rendering?
• RQ3 How does NeuS perform on complex geometries and thin structures under occlusion compared to IDR, NeRF, and UNISURF?
• RQ4 What impact do Eikonal regularization and geometric initialization have on learning accurate SDFs for multi-view reconstruction?

Key findings

• NeuS achieves superior reconstruction quality on DTU and BlendedMVS, outperforming IDR, NeRF, and UNISURF in both mask-assisted and mask-free settings.
• The proposed unbiased, occlusion-aware weight function enables accurate zero-level surface extraction from the neural SDF without foreground masks.
• NeuS accurately reconstructs thin structures and handles severe self-occlusion and abrupt depth changes.
• Ablation shows naive volume rendering or straightforward SDF-to-volume mappings introduce bias or artifacts; proper weight construction plus Eikonal regularization improves SDF accuracy.
• Qualitative and quantitative results demonstrate robust performance in challenging scenarios, including thin metallic parts and occluded regions.