[Paper Review] S3Net: A Single Stream Structure for Depth Guided Image Relighting

Hao-Hsiang Yang, Wei‐Ting Chen+1 more|arXiv (Cornell University)|May 3, 2021

Advanced Vision and Imaging36 references4 citations

TL;DR

S3Net is a single-stream deep learning framework for depth-guided any-to-any image relighting, combining source and guided images with their depth maps as input to generate relit images using an encoder-decoder architecture enhanced with attention and residual modules. It achieves 3rd place in the NTIRE 2021 challenge with a 0.6969 SSIM on the VIDIT dataset using a wavelet-based SSIM loss for improved perceptual quality.

ABSTRACT

Depth guided any-to-any image relighting aims to generate a relit image from the original image and corresponding depth maps to match the illumination setting of the given guided image and its depth map. To the best of our knowledge, this task is a new challenge that has not been addressed in the previous literature. To address this issue, we propose a deep learning-based neural Single Stream Structure network called S3Net for depth guided image relighting. This network is an encoder-decoder model. We concatenate all images and corresponding depth maps as the input and feed them into the model. The decoder part contains the attention module and the enhanced module to focus on the relighting-related regions in the guided images. Experiments performed on challenging benchmark show that the proposed model achieves the 3 rd highest SSIM in the NTIRE 2021 Depth Guided Any-to-any Relighting Challenge.

Motivation & Objective

• To address the novel challenge of depth-guided any-to-any image relighting, where a source image is relit to match the illumination of a guided image using depth maps.
• To design an efficient single-stream architecture that avoids the computational burden of dual-stream networks while effectively fusing RGB and depth features.
• To improve relighting quality by incorporating attention mechanisms and enhanced feature refinement in the decoder for directional lighting and shadow transfer.
• To optimize training using a multi-scale loss function based on discrete wavelet transform to enhance global and local image fidelity.
• To achieve state-of-the-art performance in the NTIRE 2021 Depth Guided Any-to-Any Relighting Challenge.

Proposed method

• The model uses a single-stream encoder-decoder architecture that concatenates source and guided images with their corresponding depth maps into a single 6-channel input tensor.
• The encoder employs Res2Net blocks to extract hierarchical features with multi-scale感受 (receptive field expansion).
• The decoder integrates an attention module to emphasize relighting-relevant regions and an enhanced module to refine feature maps for better shadow and lighting consistency.
• A wavelet-based SSIM loss (L_{W-SSIM}) is used during training to improve perceptual quality by capturing multi-scale structural details.
• The loss function combines L_{Cha}, L_{SSIM}, and L_{Per} for color, structural, and perceptual fidelity, with L_{W-SSIM} replacing L_{SSIM} in the best-performing setting.
• The network is trained end-to-end on the VIDIT dataset, with inference time of 2.042 seconds per 1024×1024 image.

Experimental results

Research questions

• RQ1 Can a single-stream network effectively fuse RGB and depth features for any-to-any image relighting without dual-stream complexity?
• RQ2 How does the integration of attention and enhanced modules improve relighting quality in shadow and lighting transfer?
• RQ3 To what extent does using a wavelet SSIM loss improve perceptual quality compared to standard SSIM in relighting tasks?
• RQ4 Can the model generalize across diverse lighting conditions, including color temperature and directional illumination, using only depth maps as spatial guidance?
• RQ5 What is the performance of the proposed method in comparison to existing approaches in the NTIRE 2021 benchmark?

Key findings

• The proposed S3Net achieved 3rd place in the NTIRE 2021 Depth Guided Any-to-Any Relighting Challenge in terms of SSIM and MPS, with a validation SSIM of 0.7022 and test SSIM of 0.6784.
• The model achieved a PSNR of 19.2212 and LPIPS of 0.1566 on the test set, demonstrating strong perceptual and structural fidelity.
• The ablation study showed that using depth maps (setting 2) improved SSIM from 0.6821 to 0.6913 compared to using only images (setting 1).
• Replacing standard SSIM with wavelet SSIM (setting 3) further improved SSIM to 0.6969 and PSNR to 19.1281, confirming the effectiveness of the multi-scale loss.
• Visual comparisons confirmed that the wavelet SSIM loss produces relit images closer to ground truth, with better shadow removal and lighting transfer.
• The model failed on images with large, ambiguous shadows due to limited depth map information, indicating a need for improved 3D reasoning in future work.