[Paper Review] Multi-view Gradient Consistency for SVBRDF Estimation of Complex Scenes under Natural Illumination

Alen Joy, Charalambos Poullis|arXiv (Cornell University)|Feb 25, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes an end-to-end method for estimating spatially varying BRDFs in complex, large-scale outdoor scenes under natural illumination using multi-view images. By employing a differentiable path tracer with least-square conformal mapping and a novel multi-view gradient consistency loss in a two-step optimization, the approach achieves up to 50% lower reconstruction loss and improved disentanglement of diffuse and specular reflectance compared to SOTA.

ABSTRACT

This paper presents a process for estimating the spatially varying surface reflectance of complex scenes observed under natural illumination. In contrast to previous methods, our process is not limited to scenes viewed under controlled lighting conditions but can handle complex indoor and outdoor scenes viewed under arbitrary illumination conditions. An end-to-end process uses a model of the scene's geometry and several images capturing the scene's surfaces from arbitrary viewpoints and under various natural illumination conditions. We develop a differentiable path tracer that leverages least-square conformal mapping for handling multiple disjoint objects appearing in the scene. We follow a two-step optimization process and introduce a multi-view gradient consistency loss which results in up to 30-50% improvement in the image reconstruction loss and can further achieve better disentanglement of the diffuse and specular BRDFs compared to other state-of-the-art. We demonstrate the process in real-world indoor and outdoor scenes from images in the wild and show that we can produce realistic renders consistent with actual images using the estimated reflectance properties. Experiments show that our technique produces realistic results for arbitrary outdoor scenes with complex geometry. The source code is publicly available at: https://gitlab.com/alen.joy/multi-view-gradient-consistency-for-svbrdf-estimation-of-complex-scenes-under-natural-illumination

Motivation & Objective

• To estimate spatially varying BRDFs in large-scale, complex scenes under arbitrary natural illumination, overcoming limitations of prior methods that require controlled lighting.
• To enable accurate reflectance estimation for multiple disjoint objects with complex geometry, not just single synthetic objects.
• To improve disentanglement of diffuse and specular BRDF components for better re-lighting fidelity.
• To achieve robust performance on real-world scenes from images in the wild, including challenging outdoor environments.
• To provide a reusable, differentiable framework for inverse rendering with explicit control over reflectance parameters.

Proposed method

• Uses multi-view stereo to reconstruct scene geometry from real-world images captured under natural lighting.
• Employs a differentiable path tracer based on least-square conformal mapping to handle multiple disjoint objects with complex geometry.
• Applies a two-step optimization process to separately refine diffuse and specular BRDF parameters for better disentanglement.
• Introduces a multi-view gradient consistency loss that minimizes gradient variance across views and enforces consistent per-point gradient directions.
• Parameterizes the modified Cook-Torrance BRDF to model spatially varying surface reflectance.
• Optimizes SVBRDF parameters by minimizing a combined loss of reconstruction error and gradient consistency across multiple views.

Experimental results

Research questions

• RQ1 Can SVBRDF estimation be effectively performed on large-scale, complex outdoor scenes under arbitrary natural illumination without controlled lighting?
• RQ2 How can multi-view consistency be enforced to improve the accuracy and robustness of reflectance estimation?
• RQ3 Can a differentiable renderer handle multiple disjoint objects with complex geometry while maintaining optimization stability?
• RQ4 To what extent does the proposed multi-view gradient consistency loss improve disentanglement of diffuse and specular BRDF components?
• RQ5 How does the resolution of the SVBRDF map affect the visual realism and computational cost of the reconstruction?

Key findings

• The proposed method achieves up to 50% improvement in image reconstruction loss for scenes with non-metallic surfaces compared to state-of-the-art methods.
• For metallic surfaces, the method reduces reconstruction loss by up to 10%, demonstrating robustness even in challenging cases.
• The multi-view gradient consistency loss significantly improves the disentanglement of diffuse and specular reflectance components.
• Re-lighting with novel environment maps results in up to 50% reduction in RMSE, indicating high fidelity in novel view and lighting synthesis.
• High-resolution SVBRDF maps (e.g., 2048²) are necessary to capture fine details such as hand-drawn markings on building facades.
• Lower-resolution maps (e.g., 512²) fail to preserve essential surface details, while 4096² offers only marginal gains with increased computational cost.