[Paper Review] Neural Inverse Rendering for General Reflectance Photometric Stereo

Tatsunori Taniai, Takanori Maehara|arXiv (Cornell University)|Feb 28, 2018

Computer Graphics and Visualization Techniques56 citations

TL;DR

This paper presents an unsupervised, physics-based CNN framework (PSNet and IRNet) that jointly estimates per-pixel surface normals and BRDFs from multi-illumination images, by rendering observations and minimizing reconstruction losses directly on a test scene, achieving state-of-the-art performance on real-world data without ground-truth normals or pre-training.

ABSTRACT

We present a novel convolutional neural network architecture for photometric stereo (Woodham, 1980), a problem of recovering 3D object surface normals from multiple images observed under varying illuminations. Despite its long history in computer vision, the problem still shows fundamental challenges for surfaces with unknown general reflectance properties (BRDFs). Leveraging deep neural networks to learn complicated reflectance models is promising, but studies in this direction are very limited due to difficulties in acquiring accurate ground truth for training and also in designing networks invariant to permutation of input images. In order to address these challenges, we propose a physics based unsupervised learning framework where surface normals and BRDFs are predicted by the network and fed into the rendering equation to synthesize observed images. The network weights are optimized during testing by minimizing reconstruction loss between observed and synthesized images. Thus, our learning process does not require ground truth normals or even pre-training on external images. Our method is shown to achieve the state-of-the-art performance on a challenging real-world scene benchmark.

Motivation & Objective

• Address photometric stereo with unknown, general BRDFs beyond Lambertian assumptions.
• Develop an unsupervised, per-scene learning framework that does not require ground-truth normals or external pre-training.
• Incorporate physical reflectance rendering into a CNN architecture to learn complex BRDFs.
• Ensure permutation invariance and applicability to real-world scenes by testing directly on the target data.

Proposed method

• Two-subnetwork architecture: PSNet predicts a per-pixel surface normal map from concatenated input images under known lighting directions; IRNet synthesizes each observed image via a rendering-inspired BRDF and shadow model and outputs a reflectance image.
• IRNet inputs include a specularity hint channel, a global feature blend with PSNet outputs, and per-illumination information to reconstruct each observation through the rendering equation I = R ⊙ max(ℓ^T N, 0).
• End-to-end training minimizes a reconstruction loss between observed and synthesized images (Lrec) plus an early-stage weak supervision term (Lprior) on normals to stabilize learning on complex scenes.
• Learning is performed directly on the test scene during SGD (no pre-training), with random dropout in the reconstruction loss to improve convergence, and BatchNorm configured for training statistics on the given data.
• The reconstruction loss focuses on target object regions and uses an MAE formulation to be robust to specular highlights.

Experimental results

Research questions

• RQ1 Can unsupervised, per-scene learning recover accurate surface normals for general BRDFs without ground-truth normals?
• RQ2 Does incorporating a physics-based rendering equation into a CNN enable learning of broad BRDFs beyond hand-crafted models?
• RQ3 Does early-stage weak supervision improve convergence and accuracy on complex real-world scenes compared to no supervision or full-stage supervision?
• RQ4 Is it possible to achieve state-of-the-art photometric stereo performance on real benchmarks (e.g., DiLiGenT) without pre-training on external data?

Key findings

• Achieves state-of-the-art performance on the DiLiGenT real-world benchmark, with best average score across eight scenes and best individual scores on most scenes.
• The full architecture (including the specularity input and global observation blending) yields the best accuracy; removing the specularity channel notably degrades performance on metal-like surfaces.
• Early-stage weak supervision stabilizes learning and yields better median/mean angular errors than no supervision or all-stage supervision, enabling effective optimization without ground-truth normals.
• Training is performed per scene in an unsupervised fashion, directly fitting network weights to the test data without prior pre-training, and still outperforms several prior learning-based and classical methods.
• Incorporating a specular cue and physics-based rendering into IRNet improves learning of complex BRDFs compared to purely data-driven approaches.
• The method remains computationally intensive (roughly hours per scene) but demonstrates robustness and flexibility in handling varied materials and lighting.