[Paper Review] Scene relighting with illumination estimation in the latent space on an encoder-decoder scheme

Alexandre Pierre Dherse, Martin Everaert+1 more|arXiv (Cornell University)|Jun 3, 2020

Advanced Vision and Imaging27 references8 citations

TL;DR

This paper proposes a latent-space encoder-decoder framework for scene relighting that estimates illumination conditions from a target image and transfers them to an input image, enabling realistic relighting without explicit lighting annotations. The method achieves improved consistency in color temperature and basic shading transfer, though realism and accurate shadow reconstruction remain limited, particularly on unseen scenes.

ABSTRACT

The image relighting task of transferring illumination conditions between two images offers an interesting and difficult challenge with potential applications in photography, cinematography and computer graphics. In this report we present methods that we tried to achieve that goal. Our models are trained on a rendered dataset of artificial locations with varied scene content, light source location and color temperature. With this dataset, we used a network with illumination estimation component aiming to infer and replace light conditions in the latent space representation of the concerned scenes.

Motivation & Objective

• To develop a generalizable image relighting method that transfers illumination from a target image to an input image without requiring multiple inputs or explicit lighting annotations.
• To enable automatic illumination estimation directly from the latent space representation of the target image, avoiding manual light source specification.
• To decouple scene content and illumination information in the latent space to allow flexible recombination of input scene with target illumination.
• To evaluate the model’s performance on a diverse, artificially rendered dataset (VIDIT) with varied lighting conditions and scene content.
• To identify limitations in realism and shadow reconstruction, particularly in dark regions, and suggest improvements for future work.

Proposed method

• The model uses a shared encoder-decoder architecture to process both input and target images, extracting latent representations of scene and illumination.
• Illumination estimation is performed via a fully-connected network or environment map prediction in the latent space, enabling disentangled lighting representation.
• The scene features from the input image are combined with the illumination features from the target image in the latent space before decoding.
• The network is trained with a combination of reconstruction loss and perceptual loss (LPIPS), with additional optimization for the Envmap + scene model to minimize LPIPS.
• Skip connections are used to preserve spatial details during decoding, and PixelShuffle is considered as an alternative to transposed convolutions to reduce checkerboard artifacts.
• A conditional GAN or relativistic GAN is suggested as a future enhancement to improve realism and consistency of relit images.

Experimental results

Research questions

• RQ1 Can a single-image-based relighting method effectively estimate and transfer illumination conditions from a target image to an input image in the latent space?
• RQ2 To what extent can a disentangled latent representation of scene and illumination improve relighting quality and generalization across diverse scenes?
• RQ3 Why do models fail to accurately reconstruct shadows and deep dark areas, and how does this affect realism?
• RQ4 How does automatic illumination estimation compare to methods requiring explicit lighting input or multiple views of the same scene?
• RQ5 What improvements can be made to enhance realism and consistency of relit images, especially in regions with strong shadows?

Key findings

• The Envmap + scene model achieved the best LPIPS score (0.2564), indicating superior perceptual similarity to ground truth, though it was specifically optimized for this metric.
• The illumination predictor model achieved the highest SSIM (0.3365), suggesting better structural preservation of image content.
• The SNR values were relatively low (18.11–18.66 dB), indicating limited signal-to-noise performance, especially in dark regions.
• The models showed strong performance in color temperature transfer but struggled with accurate directional lighting changes, particularly in shadowed areas.
• The network often removed deep shadows and replaced them with uniform colors, indicating a failure to 'imagine' plausible content in dark regions.
• Generalization to unseen scenes was limited, as performance dropped significantly on the validation set compared to the training set, suggesting poor zero-shot transfer capability.