[Paper Review] Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images

Sai Bi, Zexiang Xu+5 more|arXiv (Cornell University)|Jul 20, 2020

Computer Graphics and Visualization Techniques6 citations

TL;DR

This paper proposes Deep Reflectance Volumes, a deep learning framework that reconstructs high-fidelity 3D scene geometry and appearance from unstructured multi-view flash images captured with a mobile phone. By learning volumetric representations of opacity, surface normals, and spatially-varying reflectance (albedo, roughness), and using a differentiable, physically-based ray-marching renderer, the method enables photorealistic novel view synthesis and relighting under arbitrary lighting, outperforming mesh-based baselines and supporting intuitive material editing.

ABSTRACT

We present a deep learning approach to reconstruct scene appearance from unstructured images captured under collocated point lighting. At the heart of Deep Reflectance Volumes is a novel volumetric scene representation consisting of opacity, surface normal and reflectance voxel grids. We present a novel physically-based differentiable volume ray marching framework to render these scene volumes under arbitrary viewpoint and lighting. This allows us to optimize the scene volumes to minimize the error between their rendered images and the captured images. Our method is able to reconstruct real scenes with challenging non-Lambertian reflectance and complex geometry with occlusions and shadowing. Moreover, it accurately generalizes to novel viewpoints and lighting, including non-collocated lighting, rendering photorealistic images that are significantly better than state-of-the-art mesh-based methods. We also show that our learned reflectance volumes are editable, allowing for modifying the materials of the captured scenes.

Motivation & Objective

• To enable high-quality, practical 3D scene reconstruction and relighting using unstructured, multi-view flash images captured with consumer-grade devices.
• To address limitations of mesh-based reconstruction in non-Lambertian and textureless regions by learning a volumetric representation of scene geometry and reflectance.
• To support novel view synthesis and relighting under arbitrary lighting and viewpoints, including non-collocated lighting setups.
• To enable intuitive material editing by disentangling geometry and reflectance in the learned volume representation.
• To demonstrate that deep priors in a volumetric network improve optimization stability and generalization compared to direct voxel optimization.

Proposed method

• The method learns a 3D volumetric representation consisting of opacity, surface normal, albedo, and roughness volumes from multi-view flash images.
• A deep neural network decodes scene geometry and reflectance from a learned encoding vector, with network parameters optimized per-scene.
• A differentiable, physically-based volume ray marching renderer computes shading using local normals and reflectance, accumulating color and opacity along rays from both view and light directions.
• Opacity is learned globally to enable accurate hard shadow rendering under novel lighting, even when shadows were not present in training images.
• The reflectance volumes model spatially-varying, view- and light-dependent effects such as specularities and glossiness, unlike prior methods that use single-color voxels.
• Optimization minimizes pixel-wise reconstruction loss between rendered and captured images during training, with inference enabling free viewpoint and lighting control.

Experimental results

Research questions

• RQ1 Can a deep learning framework reconstruct complex real scenes with non-Lambertian reflectance and occlusions from unstructured mobile phone flash images?
• RQ2 Can a volumetric representation with learned opacity, normals, and reflectance enable accurate relighting under arbitrary lighting and viewpoints?
• RQ3 Does a differentiable, physically-based volume rendering framework outperform mesh-based methods in view synthesis and relighting quality?
• RQ4 Can the learned reflectance volumes support intuitive material editing without altering geometry or color?
• RQ5 Does incorporating deep priors improve optimization convergence and generalization compared to direct voxel optimization?

Key findings

• The method achieves significantly better photorealistic rendering quality than state-of-the-art mesh-based methods, especially in handling specularities, shadows, and non-Lambertian materials.
• With 200 training images, the method achieves comparable PSNR and SSIM to the full set of 385 images, indicating robustness to moderate input reduction.
• Performance degrades noticeably with fewer than 100 training images due to insufficient geometric and photometric constraints.
• The deep prior-based optimization converges significantly faster than direct voxel optimization, demonstrating the effectiveness of inductive bias from the network architecture.
• The method successfully enables material editing by modifying the roughness volume, producing realistic glossy effects not present in the original scene.
• The learned reflectance volumes disentangle geometry and reflectance, allowing component-specific editing without affecting other scene properties.