[Paper Review] Deep Relighting Networks for Image Light Source Manipulation

Wang, Li-Wen, Wan-Chi Siu+3 more|arXiv (Cornell University)|Aug 19, 2020

Advanced Vision and Imaging37 references4 citations

TL;DR

This paper proposes a Deep Relighting Network (DRN) that formulates single-image relighting as a three-stage process: scene reconversion via a back-projection-augmented autoencoder, adversarial shadow prior estimation for new light directions, and re-rendering to generate images under target illumination. The method achieves state-of-the-art performance, winning the best PSNR in the AIM2020 Image Relighting Challenge.

ABSTRACT

Manipulating the light source of given images is an interesting task and useful in various applications, including photography and cinematography. Existing methods usually require additional information like the geometric structure of the scene, which may not be available for most images. In this paper, we formulate the single image relighting task and propose a novel Deep Relighting Network (DRN) with three parts: 1) scene reconversion, which aims to reveal the primary scene structure through a deep auto-encoder network, 2) shadow prior estimation, to predict light effect from the new light direction through adversarial learning, and 3) re-renderer, to combine the primary structure with the reconstructed shadow view to form the required estimation under the target light source. Experimental results show that the proposed method outperforms other possible methods, both qualitatively and quantitatively. Specifically, the proposed DRN has achieved the best PSNR in the "AIM2020 - Any to one relighting challenge" of the 2020 ECCV conference.

Motivation & Objective

• To address the challenge of single-image relighting without requiring geometric scene priors or explicit 3D structure.
• To enable manipulation of both light direction and color temperature in a single image, producing realistic shadows and illumination effects.
• To improve image quality and structural consistency in relighting by decoupling scene structure from light effects.
• To overcome the difficulty of training on implicit light effect supervision by introducing a shadow-region discriminator.

Proposed method

• The DRN uses a deep auto-encoder with back-projection layers to reconstruct the primary scene structure from a single image, preserving spatial details across down- and up-sampling.
• A shadow prior estimation module uses adversarial learning with a novel shadow-region discriminator to predict realistic light effects from a new light source direction and color temperature.
• The re-renderer combines the reconstructed scene structure with the predicted shadow prior to generate the final image under the target light source.
• The network is trained end-to-end with a perceptual loss and adversarial loss, enabling high-fidelity relighting without ground-truth light effects.
• Back-projection theory is integrated into the auto-encoder to stabilize feature learning and improve reconstruction fidelity.
• The method is specifically designed for the 'any to one' relighting task, where any input illumination is transformed to a specific target light setting.

Experimental results

Research questions

• RQ1 Can a deep learning model generate realistic relighting effects without requiring geometric priors or 3D scene information?
• RQ2 How can shadow effects be accurately predicted and rendered under new light sources when no explicit supervision is available?
• RQ3 Can back-projection theory enhance feature reconstruction in auto-encoder architectures for better relighting performance?
• RQ4 To what extent can adversarial learning improve the realism of relit images when light effects are difficult to measure directly?
• RQ5 Can a unified network architecture generalize across diverse light source configurations (direction and color temperature) in a single-image setting?

Key findings

• The proposed DRN achieved the highest PSNR of 17.59 dB on the VIDIT validation dataset, outperforming U-Net, Retinex-Net, and Pix2Pix.
• In the AIM2020 Image Relighting Challenge (Track 1: any to one), the DRN achieved the best PSNR score in the final testing phase.
• Visual comparisons show that DRN correctly manipulates both light direction and color temperature, while minimizing artifacts and preserving shadow consistency.
• The method successfully recasts shadows in challenging cases, such as nearly black input images, outperforming U-Net and Pix2Pix in both structure recovery and shadow realism.
• The shadow-region discriminator significantly improves training stability and realism, especially in low-light or high-contrast scenarios.
• Despite strong performance, the network struggles to recover fine structural details when the input image has lost most information (e.g., completely dark regions), indicating a limitation in inpainting capability.