[Paper Review] DeLTra: Deep Light Transport for Projector-Camera Systems

Bingyao Huang, Haibin Ling|arXiv (Cornell University)|Mar 6, 2020

Advanced Vision and Imaging49 references4 citations

TL;DR

DeLTra is an end-to-end trainable deep learning framework for radiometrically uncalibrated projector-camera light transport, combining a DepthToAttribute module for ray/depth/normal estimation and a CNN-based ShadingNet for photorealistic rendering. It introduces a differentiable direct light mask using epipolar constraints to handle occlusion, enabling unified application to image-based relighting, projector compensation, and depth/normal reconstruction with state-of-the-art performance.

ABSTRACT

In projector-camera systems, light transport models the propagation from projector emitted radiance to camera-captured irradiance. In this paper, we propose the first end-to-end trainable solution named Deep Light Transport (DeLTra) that estimates radiometrically uncalibrated projector-camera light transport. DeLTra is designed to have two modules: DepthToAtrribute and ShadingNet. DepthToAtrribute explicitly learns rays, depth and normal, and then estimates rough Phong illuminations. Afterwards, the CNN-based ShadingNet renders photorealistic camera-captured image using estimated shading attributes and rough Phong illuminations. A particular challenge addressed by DeLTra is occlusion, for which we exploit epipolar constraint and propose a novel differentiable direct light mask. Thus, it can be learned end-to-end along with the other DeLTra modules. Once trained, DeLTra can be applied simultaneously to three projector-camera tasks: image-based relighting, projector compensation and depth/normal reconstruction. In our experiments, DeLTra shows clear advantages over previous arts with promising quality and meanwhile being practically convenient.

Motivation & Objective

• Address the challenge of radiometrically uncalibrated light transport in projector-camera systems, where calibration is impractical or error-prone.
• Overcome the difficulty of modeling occlusions in light transport estimation without relying on explicit 3D geometry.
• Develop an end-to-end trainable framework that jointly optimizes depth, normal, and shading attributes for accurate image synthesis.
• Enable unified application of the model to three key projector-camera tasks: image-based relighting, projector compensation, and depth/normal reconstruction.

Proposed method

• Propose the DepthToAttribute module to explicitly estimate rays, depth, and surface normals from input depth images, and generate rough Phong illumination models.
• Introduce a differentiable direct light mask based on epipolar geometry to model occlusions, enabling end-to-end backpropagation through the occlusion boundary.
• Design a CNN-based ShadingNet that fuses estimated shading attributes and rough Phong illuminations to render photorealistic camera-captured images.
• Train the entire DeLTra system end-to-end using a differentiable rendering loss, allowing joint optimization of depth, normal, and shading parameters.
• Leverage epipolar constraints to enforce geometric consistency in light visibility estimation, improving robustness to occlusions.
• Use a multi-task learning objective to simultaneously optimize for relighting, compensation, and reconstruction tasks during training.

Experimental results

Research questions

• RQ1 Can an end-to-end differentiable framework effectively model uncalibrated projector-camera light transport without requiring radiometric calibration?
• RQ2 How well can a learned light transport model handle occlusions in real-world projector-camera systems?
• RQ3 To what extent can a single unified model perform image-based relighting, projector compensation, and depth/normal reconstruction simultaneously?
• RQ4 Does the differentiable direct light mask based on epipolar constraints improve occlusion modeling compared to conventional methods?

Key findings

• DeLTra achieves state-of-the-art performance in image-based relighting, producing photorealistic results even under uncalibrated conditions.
• The differentiable direct light mask significantly improves occlusion handling, leading to more accurate light transport estimation.
• The model enables practical projector compensation with reduced visual artifacts compared to prior methods.
• DeLTra successfully reconstructs depth and surface normals with high fidelity, demonstrating its multi-task capability.
• The end-to-end training scheme enables joint optimization of all components, resulting in improved generalization and robustness.
• The framework is practically convenient and deployable across multiple projector-camera applications without retraining or calibration.