[Paper Review] NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination

Xiuming Zhang, Pratul P. Srinivasan+4 more|arXiv (Cornell University)|Jun 3, 2021

Advanced Vision and Imaging60 references57 citations

TL;DR

NeRFactor recovers a full 3D surface with normals, light visibility, albedo, and spatially-varying BRDFs from multi-view images under one unknown illumination, enabling relighting with shadows and material editing.

ABSTRACT

We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu/xiuming/projects/nerfactor/.

Motivation & Objective

• Recover shape, spatially-varying reflectance, and environment lighting from multi-view images captured under a single unknown illumination.
• Distill NeRF-based volume density into a surface representation to enable efficient shading and relighting.
• Introduce a data-driven BRDF prior learned from real BRDF measurements to constrain material estimation.
• Distinguish shadows from albedo to enable realistic shadowed relighting under arbitrary lighting conditions.

Proposed method

• Initialize geometry with NeRF (or MVS) to obtain surface location, normals, and light visibility.
• Re-parameterize normals and visibility as MLPs to denoise and smooth them while fitting the re-rendering loss.
• Model BRDFs as a sum of a Lambertian albedo and a learned non-diffuse component via a BRDF latent space pretrained on real BRDFs (MERL) and decoded per surface point.
• Represent lighting with a high-frequency HDR light probe and optimize under a direct, single-bounce rendering assumption.
• Train with a reconstruction loss plus smoothness and prior losses on normals, visibility, albedo, and BRDFs to disentangle shape, reflectance, and illumination.

Experimental results

Research questions

• RQ1 Can we factorize appearance into geometry, spatially-varying reflectance, and unknown environment lighting from images captured under a single illumination?
• RQ2 Does distilling NeRF-based geometry into a surface representation with learned priors enable accurate relighting and material editing?
• RQ3 How well can a data-driven BRDF prior support plausible, spatially-varying materials in a one-light under-constrained setting?

Key findings

• The method recovers plausible 3D surface normals, light visibility, albedo, and BRDFs without supervision from multi-view images with unknown lighting.
• Explicit modeling of light visibility enables separation of shadows from albedo and realistic shadowing under novel lighting.
• A NeRF-derived initialization plus learned priors improves geometry and material estimates, enabling free-viewpoint relighting and material editing.
• Results show NeRFactor outperforms classic and deep-learning state-of-the-art on multiple tasks without requiring additional observed lighting conditions.