[Paper Review] Radiometric Scene Decomposition: Scene Reflectance, Illumination, and Geometry from RGB-D Images

Stephen Lombardi, Ko Nishino|arXiv (Cornell University)|Apr 5, 2016

3D Surveying and Cultural Heritage10 references3 citations

TL;DR

This paper presents a novel Bayesian framework for radiometric scene decomposition from RGB-D images, jointly estimating spatially-varying reflectance, natural illumination, and refined geometry while modeling interreflection, shadows, and diffuse texture. The method achieves accurate appearance prediction from novel views and enables full scene understanding by resolving ambiguities through structured priors and iterative optimization on real-world scenes with complex materials and lighting.

ABSTRACT

Recovering the radiometric properties of a scene (i.e., the reflectance, illumination, and geometry) is a long-sought ability of computer vision that can provide invaluable information for a wide range of applications. Deciphering the radiometric ingredients from the appearance of a real-world scene, as opposed to a single isolated object, is particularly challenging as it generally consists of various objects with different material compositions exhibiting complex reflectance and light interactions that are also part of the illumination. We introduce the first method for radiometric scene decomposition that handles those intricacies. We use RGB-D images to bootstrap geometry recovery and simultaneously recover the complex reflectance and natural illumination while refining the noisy initial geometry and segmenting the scene into different material regions. Most important, we handle real-world scenes consisting of multiple objects of unknown materials, which necessitates the modeling of spatially-varying complex reflectance, natural illumination, texture, interreflection and shadows. We systematically evaluate the effectiveness of our method on synthetic scenes and demonstrate its application to real-world scenes. The results show that rich radiometric information can be recovered from RGB-D images and demonstrate a new role RGB-D sensors can play for general scene understanding tasks.

Motivation & Objective

• To address the challenge of recovering radiometric properties—reflectance, illumination, and geometry—from real-world scenes with complex materials and lighting interactions.
• To overcome limitations of prior methods that assume Lambertian reflectance and point light sources in single-object settings.
• To enable robust decomposition in scenes with multiple objects of unknown, spatially-varying materials, interreflection, shadows, and non-Lambertian effects.
• To leverage RGB-D data not just for geometry, but as a foundation for full radiometric scene understanding.
• To develop a systematic, differentiable framework that enables maximum a posteriori (MAP) estimation with ordered variable optimization and geometric refinement.

Proposed method

• Models spatially-varying reflectance by decoupling isotropic BRDFs from diffuse texture, enabling compact representation of complex surface materials.
• Uses path tracing to simulate indirect illumination and interreflection in the image formation likelihood, capturing non-local light transport.
• Introduces a Bayesian formulation with structured priors to resolve ambiguities among reflectance, illumination, and geometry.
• Employs geometric bases on the scene surface to refine low-frequency errors in RGB-D geometry with a small set of parameters.
• Performs ordered optimization: first segmenting the scene and estimating reflectance/illumination, then incorporating texture, and finally refining geometry.
• Derives gradients of the log posterior for tractable MAP estimation, enabling end-to-end optimization of all radiometric components.

Experimental results

Research questions

• RQ1 Can RGB-D data be used to jointly recover reflectance, illumination, and geometry in real-world scenes with complex materials and lighting interactions?
• RQ2 How can spatially-varying reflectance and diffuse texture be modeled effectively while avoiding overfitting?
• RQ3 To what extent can interreflection and shadows be accurately simulated and estimated using path tracing within a Bayesian optimization framework?
• RQ4 Can geometry refinement from RGB-D data improve the accuracy of reflectance and illumination estimation in the presence of sensor noise and oversmoothing?
• RQ5 How does the ordered optimization strategy prevent artifacts such as texture modeling specular highlights?

Key findings

• The method achieves accurate appearance prediction from novel views in both synthetic and real-world scenes, demonstrating robustness to geometric perturbations and complex lighting.
• On synthetic scenes, the framework recovers reflectance and illumination with high accuracy, even when geometry is artificially perturbed.
• In real-world scenes, the method correctly identifies specular reflections (e.g., on tables) as material properties rather than texture changes, avoiding misclassification.
• The framework successfully segments scenes into material regions and recovers fine texture details, such as text on books and mugs, despite sensor noise and limited viewpoints.
• Geometry refinement significantly improves shape accuracy, especially for complex features like mug handles, when optimized after reflectance and illumination.
• The staged optimization process prevents overfitting—e.g., texture does not model specular highlights—demonstrating the necessity of the ordered variable update strategy.