[Paper Review] A CNN Based Approach for the Near-Field Photometric Stereo Problem

Fotios Logothetis, Ignas Budvytis+2 more|Cambridge University Engineering Department Publications Database|Sep 12, 2020

Optical measurement and interference techniques1 references4 citations

TL;DR

This paper presents the first CNN-based approach for near-field photometric stereo that handles realistic lighting effects such as radial light attenuation, perspective geometry, and specular reflections. By iteratively estimating surface normals via a per-pixel CNN and refining depth through numerical integration, the method achieves state-of-the-art accuracy, reducing average angular error by over 6° on synthetic data compared to prior methods.

ABSTRACT

Reconstructing the 3D shape of an object using several images under different light sources is a very challenging task, especially when realistic assumptions such as light propagation and attenuation, perspective viewing geometry and specular light reflection are considered. Many of works tackling Photometric Stereo (PS) problems often relax most of the aforementioned assumptions. Especially they ignore specular reflection and global illumination effects. In this work, we propose the first CNN based approach capable of handling these realistic assumptions in Photometric Stereo. We leverage recent improvements of deep neural networks for far-field Photometric Stereo and adapt them to near field setup. We achieve this by employing an iterative procedure for shape estimation which has two main steps. Firstly we train a per-pixel CNN to predict surface normals from reflectance samples. Secondly, we compute the depth by integrating the normal field in order to iteratively estimate light directions and attenuation which is used to compensate the input images to compute reflectance samples for the next iteration. To the best of our knowledge this is the first near-field framework which is able to accurately predict 3D shape from highly specular objects. Our method outperforms competing state-of-the-art near-field Photometric Stereo approaches on both synthetic and real experiments.

Motivation & Objective

• To address the challenge of 3D shape reconstruction from near-field photometric stereo under realistic physical assumptions, including perspective geometry, point light sources, and specular reflections.
• To overcome the limitations of existing methods that simplify or ignore non-diffuse reflectance and light attenuation in near-field setups.
• To enable deep learning-based normal prediction in near-field scenarios by compensating for light attenuation using iterative depth estimation.
• To improve robustness to global illumination effects such as shadows and interreflections through data augmentation during training.
• To demonstrate superior performance on both synthetic and real-world near-field datasets compared to state-of-the-art optimization-based and CNN-based approaches.

Proposed method

• A per-pixel CNN is trained to predict surface normals from reflectance samples derived from input images after compensating for light attenuation using an estimated depth map.
• The method employs an iterative procedure: first, predict normals from compensated reflectance samples; second, integrate normals to update depth and refine light direction/attenuation estimates for the next iteration.
• Light attenuation is compensated using an initial depth estimate, enabling the use of far-field-style CNNs in a near-field context.
• Training data is synthesized using a virtual near-field capture setup, allowing for geometric data augmentation such as depth variation to reduce distribution shift.
• Observation maps are constructed from BRDF samples (MERL and Disney BSDF) and augmented with shadow and self-reflection effects to improve generalization.
• Numerical integration of the normal field is used to reconstruct depth, and the process is iterated to refine light direction and attenuation parameters.

Experimental results

Research questions

• RQ1 Can a deep CNN be effectively adapted to the near-field photometric stereo problem while handling realistic lighting and material effects?
• RQ2 How can light attenuation and perspective effects in near-field setups be compensated to enable the use of far-field CNN architectures?
• RQ3 To what extent can data augmentation strategies improve generalization to complex global illumination effects like shadows and interreflections in near-field PS?
• RQ4 Does iterative refinement of depth and light parameters improve reconstruction accuracy compared to single-step CNN inference?
• RQ5 Can a CNN-based approach outperform state-of-the-art optimization-based methods in near-field photometric stereo, especially on highly specular objects?

Key findings

• The proposed method reduces the average angular error by more than 6° on the synthetic dataset compared to the previous state-of-the-art [Quéau et al., 2018].
• The method outperforms both optimization-based approaches [Logothetis et al., 2017; Quéau et al., 2018] and naive adaptation of far-field CNNs on both synthetic and real near-field datasets.
• Qualitative results show no visible deformation on specular highlights (e.g., middle of elephant) or shadowed regions (e.g., belly of frog), unlike competing methods.
• The use of realistic BRDFs (MERL and Disney) and augmentation for shadows and self-reflections significantly improves robustness to global illumination effects.
• Naive application of far-field CNNs to near-field images without attenuation compensation results in severe distortions and higher MAE, highlighting the necessity of the iterative compensation framework.
• Iterative integration of normals leads to improved light direction and attenuation estimates, which in turn enhances normal prediction accuracy across iterations.