[Paper Review] Non-Lambertian Surface Shape and Reflectance Reconstruction Using Concentric Multi-Spectral Light Field

Mingyuan Zhou, Yu Ji+4 more|arXiv (Cornell University)|Apr 9, 2019

Advanced Vision and Imaging41 references4 citations

TL;DR

This paper proposes a concentric multi-spectral light field (CMSLF) system that enables single-shot recovery of non-Lambertian surface shape and multi-spectral reflectance without surface priors. By using spectrally multiplexed cameras and lighting on concentric rings, it exploits unique specular variation patterns across views to robustly estimate depth and separate diffuse from specular components via a physical-based dichromatic Phong model on S-Cam representations, achieving sub-2° normal error and high-fidelity reflectance reconstruction in synthetic and real-world scenes.

ABSTRACT

Recovering the shape and reflectance of non-Lambertian surfaces remains a challenging problem in computer vision since the view-dependent appearance invalidates traditional photo-consistency constraint. In this paper, we introduce a novel concentric multi-spectral light field (CMSLF) design that is able to recover the shape and reflectance of surfaces with arbitrary material in one shot. Our CMSLF system consists of an array of cameras arranged on concentric circles where each ring captures a specific spectrum. Coupled with a multi-spectral ring light, we are able to sample viewpoint and lighting variations in a single shot via spectral multiplexing. We further show that such concentric camera/light setting results in a unique pattern of specular changes across views that enables robust depth estimation. We formulate a physical-based reflectance model on CMSLF to estimate depth and multi-spectral reflectance map without imposing any surface prior. Extensive synthetic and real experiments show that our method outperforms state-of-the-art light field-based techniques, especially in non-Lambertian scenes.

Motivation & Objective

• Address the challenge of reconstructing non-Lambertian surface geometry and reflectance due to view-dependent appearance that invalidates traditional photo-consistency constraints.
• Overcome limitations of existing light field methods that require surface priors such as smoothness or polynomial shape for non-Lambertian scenes.
• Enable single-shot, high-accuracy reconstruction of both 3D shape and multi-spectral reflectance for arbitrary materials using a novel hardware design.
• Leverage unique specular variation patterns across views in a concentric camera/light configuration to enable robust depth estimation without prior assumptions.

Proposed method

• Design a concentric multi-spectral light field (CMSLF) system with cameras and a ring light arranged on concentric circles, each capturing at a specific narrowband spectrum to enable spectral multiplexing of viewpoints and lighting directions.
• Utilize surface camera (S-Cam) representation to model angular reflectance distribution from scene points back to captured light fields, enabling physical-based reflectance modeling.
• Formulate a dichromatic Phong reflectance model on S-Cam data that separates diffuse and specular components by exploiting their distinct spectral and angular characteristics.
• Initiate depth estimation using the unique specular variation pattern across views in the concentric setup, which enables robust separation of specular highlights.
• Remove specular components from all surface points using the identified characteristics, enabling specular-free S-Cam data for joint normal and reflectance estimation.
• Perform iterative refinement of surface normals and multi-spectral reflectance coefficients using specular-free S-Cam data to improve reconstruction accuracy.

Experimental results

Research questions

• RQ1 Can a concentric multi-spectral light field design enable single-shot, prior-free recovery of non-Lambertian surface shape and reflectance?
• RQ2 How can the unique specular variation pattern across views in a concentric camera configuration be exploited for robust depth estimation in non-Lambertian scenes?
• RQ3 Can a physical-based reflectance model on S-Cam representations effectively separate diffuse and specular components without surface priors?
• RQ4 To what extent does spectral multiplexing via narrowband filters enable interference-free sampling of multiple viewpoints and lighting directions in one acquisition?

Key findings

• The proposed CMSLF system achieves sub-2° normal error in synthetic sphere scenes with arbitrary material, demonstrating high geometric accuracy.
• In complex scenes such as a buddha head and jadeware, the method achieves maximum normal errors below 3°, indicating robustness to complex geometry.
• The algorithm successfully recovers high-frequency surface details and spectral reflectance maps, outperforming state-of-the-art light field methods in non-Lambertian reconstruction.
• Spectral multiplexing via narrowband filters enables interference-free sampling of 12 viewpoints and 12 lighting directions in a single capture, enabling efficient single-shot acquisition.
• The unique specular variation pattern in the concentric setup enables reliable depth estimation and specular component removal without requiring surface smoothness or shape priors.
• Real-world experiments validate the method on diverse materials including plastic and ceramic, though performance degrades under large occlusions and shadows due to insufficient spectral sampling.