[Paper Review] Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Lior Yariv, Yoni Kasten+5 more|arXiv (Cornell University)|Mar 22, 2020

Computer Graphics and Visualization Techniques345 citations

TL;DR

The paper presents IDR, an end-to-end neural architecture that jointly learns geometry (as a neural implicit surface), appearance, and camera parameters to achieve high-fidelity multiview 3D reconstructions from masked 2D images under varying lighting and materials.

ABSTRACT

In this work we address the challenging problem of multiview 3D surface reconstruction. We introduce a neural network architecture that simultaneously learns the unknown geometry, camera parameters, and a neural renderer that approximates the light reflected from the surface towards the camera. The geometry is represented as a zero level-set of a neural network, while the neural renderer, derived from the rendering equation, is capable of (implicitly) modeling a wide set of lighting conditions and materials. We trained our network on real world 2D images of objects with different material properties, lighting conditions, and noisy camera initializations from the DTU MVS dataset. We found our model to produce state of the art 3D surface reconstructions with high fidelity, resolution and detail.

Motivation & Objective

• Reconstruct 3D geometry from masked 2D images with rough or unknown camera poses without extra supervision.
• Represent geometry as a neural zero level-set (SDF) and learn a neural renderer for appearance under varied lighting.
• Enable disentanglement of geometry and appearance by conditioning the renderer on surface normals and viewing directions.
• Handle global illumination effects by introducing a global geometry feature vector to inform rendering.

Proposed method

• Represent geometry as the zero level set of a neural network f, interpreted as a signed distance function (SDF).
• Use an implicit differentiable renderer M to model the surface light field as a function of the surface point, normal, viewing direction, and a global geometry feature vector.
• Compute pixel colors via a differentiable forward model Lp(θ,γ,τ)=M(x̂p, n̂p, ẑp, vp; γ) and train parameters θ (geometry), γ (appearance), and τ (cameras) end-to-end.
• Incorporate a differentiable intersection X̂(θ,τ) between rays and the surface using a neural-unified representation that preserves exact first-derivative behavior for optimization.
• Employ a mask-based loss to leverage binary segmentation and an implicit geometric regularization (IGR) to encourage plausible surfaces.
• Augment the renderer with high-frequency input maps and a global feature vector to capture indirect lighting and scene-wide effects.

Experimental results

Research questions

• RQ1 Can an end-to-end differentiable pipeline jointly recover geometry, appearance, and camera parameters from masked 2D images?
• RQ2 Is a neural renderer that incorporates surface normals and viewing directions capable of disentangling geometry from appearance across diverse materials and lighting?
• RQ3 To what extent do global lighting effects and near-field interactions influence the quality of multiview surface reconstructions?
• RQ4 How does IDR compare to established baselines on standard multi-view datasets when cameras are known or jointly learned?

Key findings

• IDR achieves state-of-the-art or competitive 3D surface reconstructions on the DTU MVS dataset in terms of Chamfer distance and PSNR for both fixed and learned camera setups.
• Incorporating surface normals into the neural renderer is necessary for proper geometry–appearance disentanglement and leads to better geometry fidelity and rendering quality.
• The model can improve camera estimates while reconstructing watertight surfaces, outperforming baselines in camera accuracy on the Fountain dataset under unknown camera conditions.
• Ablation shows removing viewing direction, normal, or global feature vector degrades reconstruction quality, highlighting the importance of each component for accurate rendering and geometry.