[Paper Review] Multiview Neural Surface Reconstruction with Implicit Lighting and Material

Lior Yariv, Yoni Kasten+5 more|arXiv (Cornell University)|Mar 22, 2020

Computer Graphics and Visualization Techniques35 references11 citations

TL;DR

This paper proposes a neural network that jointly learns 3D surface geometry, camera parameters, and a neural renderer capable of implicitly modeling diverse lighting and materials. By representing geometry as a zero-level set and using a differentiable renderer derived from the rendering equation, the method achieves state-of-the-art 3D reconstructions from multiview images with high fidelity and detail on the DTU MVS dataset.

ABSTRACT

In this work we address the challenging problem of multiview 3D surface reconstruction. We introduce a neural network architecture that simultaneously learns the unknown geometry, camera parameters, and a neural renderer that approximates the light reflected from the surface towards the camera. The geometry is represented as a zero level-set of a neural network, while the neural renderer, derived from the rendering equation, is capable of (implicitly) modeling a wide set of lighting conditions and materials. We trained our network on real world 2D images of objects with different material properties, lighting conditions, and noisy camera initializations from the DTU MVS dataset. We found our model to produce state of the art 3D surface reconstructions with high fidelity, resolution and detail.

Motivation & Objective

• To address the challenge of multiview 3D surface reconstruction under varying lighting and material conditions.
• To jointly optimize unknown geometry, camera parameters, and a neural renderer in an end-to-end differentiable framework.
• To enable high-fidelity 3D reconstruction from real-world images with noisy camera initializations.
• To implicitly model diverse lighting conditions and material properties without explicit prior specification.

Proposed method

• Geometry is represented as the zero-level set of a neural network, enabling implicit surface reconstruction.
• A neural renderer is derived from the rendering equation to model light transport from surface to camera.
• The network is trained end-to-end on multiview images from the DTU MVS dataset with varying materials and lighting.
• Camera parameters are optimized jointly with geometry and material representations during training.
• The method implicitly models material properties and lighting through learned neural representations.
• Differentiable rendering enables backpropagation through the rendering process for joint optimization.

Experimental results

Research questions

• RQ1 Can a single neural network jointly optimize geometry, camera parameters, and a neural renderer for multiview 3D reconstruction?
• RQ2 How well can implicit modeling of lighting and materials improve reconstruction fidelity compared to explicit or fixed models?
• RQ3 To what extent can the method handle noisy camera initializations and real-world image variations?
• RQ4 Does the implicit representation of materials and lighting lead to more realistic and detailed reconstructions?
• RQ5 How does the method compare to state-of-the-art approaches on benchmark datasets like DTU MVS?

Key findings

• The method achieves state-of-the-art 3D surface reconstruction quality on the DTU MVS dataset.
• High-fidelity, high-resolution reconstructions are produced even with noisy camera initializations.
• The model effectively generalizes across diverse material properties and lighting conditions.
• The implicit neural renderer enables accurate simulation of complex light-surface interactions.
• The joint optimization of geometry, cameras, and rendering leads to improved reconstruction detail and accuracy.