[Paper Review] DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination

Stamatios Georgoulis, Konstantinos Rematas+4 more|arXiv (Cornell University)|Mar 27, 2016

Computer Graphics and Visualization Techniques1 references17 citations

TL;DR

DeLight-Net proposes a deep learning approach using a Convolutional Neural Network (CNN) to decompose reflectance maps—single 2D images of a sphere under natural illumination—into specular material parameters (Phong model) and high dynamic range (HDR) or low dynamic range (LDR) illumination maps. Trained exclusively on synthetic data, it achieves state-of-the-art performance in re-lighting and material reassignment tasks, outperforming prior methods by up to threefold in material change metrics and demonstrating robust generalization to real-world images.

ABSTRACT

In this paper we are extracting surface reflectance and natural environmental illumination from a reflectance map, i.e. from a single 2D image of a sphere of one material under one illumination. This is a notoriously difficult problem, yet key to various re-rendering applications. With the recent advances in estimating reflectance maps from 2D images their further decomposition has become increasingly relevant. To this end, we propose a Convolutional Neural Network (CNN) architecture to reconstruct both material parameters (i.e. Phong) as well as illumination (i.e. high-resolution spherical illumination maps), that is solely trained on synthetic data. We demonstrate that decomposition of synthetic as well as real photographs of reflectance maps, both in High Dynamic Range (HDR), and, for the first time, on Low Dynamic Range (LDR) as well. Results are compared to previous approaches quantitatively as well as qualitatively in terms of re-renderings where illumination, material, view or shape are changed.

Motivation & Objective

• To address the challenging inverse problem of decomposing a reflectance map into its underlying material and illumination components.
• To enable high-fidelity re-rendering by independently manipulating illumination, material, or shape after decomposition.
• To develop a data-driven method that generalizes across synthetic and real reflectance maps, including LDR images, without requiring real-world annotations.
• To improve upon existing intrinsic image decomposition methods by leveraging deep learning to model complex natural illumination and specular reflectance statistics.
• To demonstrate that a CNN trained solely on synthetic data can generalize effectively to real photographs of reflectance maps.

Proposed method

• A two-branch CNN architecture is used to jointly predict material parameters (Phong model: diffuse, specular, and roughness) and high-resolution spherical illumination maps from a single reflectance map image.
• The network is trained exclusively on a large-scale synthetic dataset of rendered reflectance maps under diverse natural illuminations and materials.
• The training data includes both HDR and LDR versions of reflectance maps, with ground truth material and illumination labels.
• The model leverages spatial inductive biases from convolutional layers to preserve fine details in illumination (e.g., windows, sky gradients) and material properties.
• A greedy inference strategy is explored to improve inference speed and stability, though the joint architecture yields better performance.
• The method enables end-to-end re-synthesis of reflectance maps under new lighting, materials, or viewing conditions using estimated components.

Experimental results

Research questions

• RQ1 Can a deep CNN trained on synthetic data accurately decompose real reflectance maps into physically plausible material and illumination components?
• RQ2 How well does the method generalize to low dynamic range (LDR) images, where dynamic range compression may affect performance?
• RQ3 To what extent can the predicted illumination and material parameters be used to re-render scenes under new lighting, material, or shape conditions?
• RQ4 Does the network learn realistic illumination patterns (e.g., sky gradients, window highlights) without explicit supervision on such features?
• RQ5 How does the performance of the proposed method compare to state-of-the-art approaches in terms of both quantitative metrics and visual quality?

Key findings

• DeLight-Net outperforms the state-of-the-art method (LN NP [37]) by a large margin in MSE and DSSIM metrics on both HDR and LDR data in the re-synthesis evaluation.
• The method achieves a three-fold improvement in material change re-rendering quality compared to the baseline, indicating strong generalization to material manipulation.
• Surprisingly, LDR-based illumination estimation yields lower MSE than HDR-based estimation, possibly due to exposure-induced whitening reducing overfitting to extreme pixel values.
• The joint CNN architecture (Joint.) achieves the best performance across all metrics, with MSE of 0.933 (HDR) and 0.950 (LDR) on mirror material re-synthesis, outperforming independent and greedy variants.
• Visual results demonstrate that the model reconstructs fine illumination details such as sky gradients and window highlights, and produces plausible re-renderings under new materials and illuminations.
• The method generalizes effectively to real reflectance maps, as validated on a custom dataset of 25 materials under 4 natural illuminations with ground-truth scans.