[Paper Review] ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal

Lanqing Guo, Chong Wang+5 more|arXiv (Cornell University)|Dec 9, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

ShadowDiffusion proposes the first diffusion-based framework for shadow removal by integrating degradation priors and generative priors through a dynamic mask-aware diffusion model (DMDM). It jointly refines the shadow-free image and shadow mask via an unrolling-inspired sampling strategy, achieving state-of-the-art performance with a 34.73dB PSNR on the SRD dataset—significantly improving over prior methods.

ABSTRACT

Recent deep learning methods have achieved promising results in image shadow removal. However, their restored images still suffer from unsatisfactory boundary artifacts, due to the lack of degradation prior embedding and the deficiency in modeling capacity. Our work addresses these issues by proposing a unified diffusion framework that integrates both the image and degradation priors for highly effective shadow removal. In detail, we first propose a shadow degradation model, which inspires us to build a novel unrolling diffusion model, dubbed ShandowDiffusion. It remarkably improves the model's capacity in shadow removal via progressively refining the desired output with both degradation prior and diffusive generative prior, which by nature can serve as a new strong baseline for image restoration. Furthermore, ShadowDiffusion progressively refines the estimated shadow mask as an auxiliary task of the diffusion generator, which leads to more accurate and robust shadow-free image generation. We conduct extensive experiments on three popular public datasets, including ISTD, ISTD+, and SRD, to validate our method's effectiveness. Compared to the state-of-the-art methods, our model achieves a significant improvement in terms of PSNR, increasing from 31.69dB to 34.73dB over SRD dataset.

Motivation & Objective

• To address boundary artifacts and residual shadows in deep learning-based shadow removal by embedding physical degradation priors.
• To overcome the limitations of existing methods that rely on inaccurate or pre-estimated shadow masks.
• To develop a unified diffusion framework that jointly optimizes shadow-free image generation and shadow mask refinement.
• To explore the potential of diffusion models in low-level vision tasks beyond denoising, particularly for image restoration with complex degradation patterns.

Proposed method

• Proposes a spatially-variant shadow degradation model that decomposes the degradation into a shadow mask and intensity map, capturing real-world lighting complexity.
• Introduces a dynamic mask-aware diffusion model (DMDM) that conditions image generation on both the input shadow image and a predicted mask, enabling joint optimization of image and mask.
• Designs an unrolling-inspired diffusive sampling strategy that explicitly integrates the degradation prior into the reverse denoising process, improving stability and convergence.
• Treats shadow mask refinement as an auxiliary task within the diffusion generator, progressively refining the mask during image restoration to reduce boundary artifacts.
• Employs a four-channel conditioning input (shadow image + mask) in the diffusion process, allowing the model to learn mask-image dependencies during training.
• Adapts the framework for low-light enhancement and exposure correction by removing mask conditioning, treating them as global degradation cases.

Experimental results

Research questions

• RQ1 Can a diffusion model effectively model the complex, spatially-variant degradation of real-world shadows when combined with physical priors?
• RQ2 How does integrating mask refinement as an auxiliary task improve the robustness and quality of shadow removal?
• RQ3 To what extent does unrolling the degradation prior into the diffusion sampling process enhance convergence and reduce artifacts?
• RQ4 Can the proposed framework generalize beyond shadow removal to other image enhancement tasks like low-light enhancement?

Key findings

• ShadowDiffusion achieves a PSNR of 34.73dB on the SRD dataset, a significant improvement from the prior state-of-the-art of 31.69dB.
• The complete model with unrolling outperforms the DMDM-only variant by 0.21dB in PSNR on the SRD dataset, demonstrating the effectiveness of the unrolling strategy.
• The unrolling framework improves sampling stability and convergence speed, reducing artifacts and instability in the final generations.
• Even without explicit mask refinement, the DMDM-only variant outperforms existing diffusion models like SR3 and WeatherDiffusion, indicating the strength of the proposed conditioning mechanism.
• The model generalizes well to low-light enhancement and exposure correction, achieving SOTA results with 27.36dB PSNR and 22.33dB PSNR respectively.
• Visual results show that ShadowDiffusion produces more natural, artifact-free outputs with sharper boundaries compared to competing methods like BMNet and Fu et al.