[Paper Review] NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

Jason Zhang, Gengshan Yang+2 more|arXiv (Cornell University)|Oct 14, 2021

Computer Graphics and Visualization TechniquesComputer Science59 references52 citations

TL;DR

NeRS proposes a surface-based neural representation that models watertight geometry with neural BRDFs, enabling robust in-the-wild sparse-view 3D reconstruction and improved novel-view synthesis over volumetric methods.

ABSTRACT

Recent history has seen a tremendous growth of work exploring implicit representations of geometry and radiance, popularized through Neural Radiance Fields (NeRF). Such works are fundamentally based on a (implicit) volumetric representation of occupancy, allowing them to model diverse scene structure including translucent objects and atmospheric obscurants. But because the vast majority of real-world scenes are composed of well-defined surfaces, we introduce a surface analog of such implicit models called Neural Reflectance Surfaces (NeRS). NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. Even more importantly, surface parameterizations allow NeRS to learn (neural) bidirectional surface reflectance functions (BRDFs) that factorize view-dependent appearance into environmental illumination, diffuse color (albedo), and specular "shininess." Finally, rather than illustrating our results on synthetic scenes or controlled in-the-lab capture, we assemble a novel dataset of multi-view images from online marketplaces for selling goods. Such "in-the-wild" multi-view image sets pose a number of challenges, including a small number of views with unknown/rough camera estimates. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering-based reconstructions. We hope that NeRS serves as a first step toward building scalable, high-quality libraries of real-world shape, materials, and illumination. The project page with code and video visualizations can be found at https://jasonyzhang.com/ners.

Motivation & Objective

• Motivate a robust alternative to volumetric NeRF-style representations for in-the-wild, sparse-view 3D reconstruction.
• Introduce Neural Reflectance Surfaces (NeRS) that enforce watertight surface geometry and factorize appearance into diffuse albedo and specular components.
• Learn shape, texture, illumination, and material properties from a small set of internet-sourced multi-view images.
• Evaluate NeRS on a novel in-the-wild Multi-view Marketplace Cars (MVMC) dataset and compare to baselines.

Proposed method

• Represent shape as a neural deformation field f_shape over the unit sphere to produce a watertight surface.
• Represent texture as a neural UV-like field f_tex over the sphere to obtain per-surface color.
• Model illumination with a neural grayscale environment map f_env and render using a Phong-based decomposition.
• Render via differentiable rasterization to compute per-pixel radiance using L_o = T(x) * I_diffuse + k_s * I_specular.
• Train with image and mask losses, a perceptual loss, and geometry regularization in a coarse-to-fine optimization schedule.

Experimental results

Research questions

• RQ1 Can a surface-based neural representation with BRDF-inspired illumination robustly recover shape, texture, and lighting from sparse, in-the-wild multi-view images?
• RQ2 How does NeRS compare to volumetric NeRF-style methods under weak camera information and limited views?
• RQ3 Does factorizing appearance into diffuse and specular components aid in disentangling illumination and material properties in the wild?
• RQ4 Can a coarse category-level template and approximate poses drive high-quality reconstructions across diverse object categories?

Key findings

• NeRS significantly outperforms baselines (NeRF*, MetaNeRF, and IDR) on MVMC across MSE, PSNR, SSIM, LPIPS, and FID.
• NeRS achieves MSE 0.0254, PSNR 16.5, SSIM 0.720, LPIPS 0.172, FID 60.9, outperforming all listed baselines.
• The surface-based representation enables accurate modeling of view-dependent reflection by disentangling environment illumination, diffuse albedo, and specular shininess.
• A coarse initialization (coarse cuboid mesh) suffices to recover detailed shape, texture, and lighting for diverse objects.
• The MVMC dataset provides large-scale, in-the-wild multi-view data to benchmark sparse-view reconstruction and novel-view synthesis.