[Paper Review] Single Image BRDF Parameter Estimation with a Conditional Adversarial Network

Mark Boss, Hendrik P. A. Lensch|arXiv (Cornell University)|Oct 11, 2019

Computer Graphics and Visualization Techniques4 citations

TL;DR

This paper proposes a conditional generative adversarial network (cGAN) that estimates spatially-varying BRDF parameters—diffuse color, normal, roughness, and metallic—from a single mobile phone flash photograph. By leveraging a multi-scale discriminator, perceptual loss, differentiable rendering, and parameter regularization, the method produces high-resolution, globally consistent material maps that closely mimic hand-authored styles while reducing artifacts from flash highlights and secondary illumination.

ABSTRACT

Creating plausible surfaces is an essential component in achieving a high degree of realism in rendering. To relieve artists, who create these surfaces in a time-consuming, manual process, automated retrieval of the spatially-varying Bidirectional Reflectance Distribution Function (SVBRDF) from a single mobile phone image is desirable. By leveraging a deep neural network, this casual capturing method can be achieved. The trained network can estimate per pixel normal, base color, metallic and roughness parameters from the Disney BRDF. The input image is taken with a mobile phone lit by the camera flash. The network is trained to compensate for environment lighting and thus learned to reduce artifacts introduced by other light sources. These losses contain a multi-scale discriminator with an additional perceptual loss, a rendering loss using a differentiable renderer, and a parameter loss. Besides the local precision, this loss formulation generates material texture maps which are globally more consistent. The network is set up as a generator network trained in an adversarial fashion to ensure that only plausible maps are produced. The estimated parameters not only reproduce the material faithfully in rendering but capture the style of hand-authored materials due to the more global loss terms compared to previous works without requiring additional post-processing. Both the resolution and the quality is improved.

Motivation & Objective

• Address the challenge of acquiring high-quality spatially-varying BRDFs from low-cost, casual captures using a single mobile phone image.
• Overcome the ill-posed nature of single-image BRDF estimation by enforcing realistic material appearance and reducing artifacts from flash and environment lighting.
• Improve upon prior methods that require post-processing or prior knowledge by learning to suppress specular artifacts and secondary illumination through a novel loss formulation.
• Enable accurate, high-resolution BRDF parameter prediction (512×512) that matches the quality needed for professional video game and film production.
• Ensure the predicted material maps reflect the stylistic consistency of hand-authored materials from tools like Substance Designer, without requiring additional refinement.

Proposed method

• Train a conditional GAN with a generator network that predicts eight per-pixel BRDF parameters: diffuse color (3-channel), normal (3-channel), roughness (1-channel), and metallic (1-channel) using the Disney BRDF model.
• Employ a multi-scale discriminator to enforce local and global consistency in predicted parameter maps, reducing artifacts from specular highlights and non-uniform lighting.
• Integrate a perceptual loss based on a pre-trained VGG network to improve visual realism and structural consistency of predicted materials.
• Use a differentiable renderer to compute a rendering loss that ensures the predicted BRDFs reproduce the input image under the same lighting conditions.
• Apply a parameter loss to regularize the predicted BRDF parameters, enforcing physical plausibility and reducing noise in the output maps.
• Train on a large synthetic dataset of 40,544 procedurally generated materials with varying environment lighting, simulating real-world mobile phone capture conditions.

Experimental results

Research questions

• RQ1 Can a deep generative model reliably estimate spatially-varying BRDF parameters from a single flash-lit mobile phone image without requiring controlled lighting or multiple views?
• RQ2 How effective is a GAN-based approach with multi-scale discriminators in reducing artifacts such as harsh flash reflections and secondary illumination compared to optimization-based or autoencoder-based methods?
• RQ3 To what extent does incorporating perceptual and rendering losses improve the visual quality and stylistic consistency of predicted BRDF maps compared to purely pixel-wise losses?
• RQ4 Can the proposed method achieve higher resolution (512×512) BRDF maps than prior work while maintaining fidelity and reducing artifacts?
• RQ5 Does the network generalize well to real-world materials despite being trained on synthetic data with controlled lighting and environment maps?

Key findings

• The proposed method achieves a 4× resolution increase over prior work, producing 512×512 BRDF parameter maps with significantly finer detail.
• The multi-scale GAN loss reduces visible artifacts from flash highlights and secondary illumination, as demonstrated in visual comparisons with Deschaintre et al. (2018) and Li et al. (2017).
• The perceptual and rendering losses lead to globally consistent parameter maps that better emulate the style of hand-authored materials from Substance Designer.
• The method successfully removes secondary illumination effects—such as light fall-off in corners—common in previous approaches, resulting in more accurate roughness and diffuse maps.
• The network generalizes well to real-world materials despite being trained exclusively on synthetic data, demonstrating robustness to uncontrolled lighting conditions.
• The inclusion of a parameter loss improves the physical plausibility of the predicted BRDFs, reducing noise and unrealistic variations in roughness and metallic values.