[Paper Review] SAM-helps-Shadow:When Segment Anything Model meet shadow removal

Xiaofeng Zhang, Chaochen Gu+1 more|arXiv (Cornell University)|Jun 1, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes SAM-helps-Shadow, a novel one-stage framework that leverages the Segment Anything Model (SAM) as a strong prior for zero-shot shadow detection and removal. By using SAM's segmentation output to guide shadow localization and employing a second-order deep unfolding network for refinement, the method achieves state-of-the-art performance on ISTD and SRD datasets with improved generalization and reduced reliance on labeled data.

ABSTRACT

The challenges surrounding the application of image shadow removal to real-world images and not just constrained datasets like ISTD/SRD have highlighted an urgent need for zero-shot learning in this field. In this study, we innovatively adapted the SAM (Segment anything model) for shadow removal by introducing SAM-helps-Shadow, effectively integrating shadow detection and removal into a single stage. Our approach utilized the model's detection results as a potent prior for facilitating shadow detection, followed by shadow removal using a second-order deep unfolding network. The source code of SAM-helps-Shadow can be obtained from https://github.com/zhangbaijin/SAM-helps-Shadow.

Motivation & Objective

• Address the critical need for zero-shot learning in real-world image shadow removal, where existing methods are limited to constrained datasets like ISTD/SRD.
• Overcome the dependency on large-scale labeled shadow datasets by leveraging SAM’s zero-shot segmentation capability for shadow detection.
• Integrate shadow detection and removal into a single, unified pipeline to improve efficiency and performance.
• Enhance generalization to real-world images by using SAM’s understanding of shadow properties without fine-tuning on shadow-specific data.
• Develop a robust, end-to-end framework that minimizes manual annotation and training complexity.

Proposed method

• Adapt the Segment Anything Model (SAM) to detect shadows by treating them as foreground objects, despite SAM’s default background classification.
• Use SAM’s segmentation mask as a strong prior to guide shadow detection, enabling zero-shot inference without retraining.
• Design a second-order deep unfolding network to iteratively refine the shadow-removed image, improving accuracy and preserving texture details.
• Integrate the SAM mask as a spatial prior in the deep unfolding network to constrain the optimization process and enhance shadow removal fidelity.
• Train the deep unfolding network in an end-to-end manner using paired real-world and shadowed images, with no need for extensive shadow annotations.
• Leverage the composability of SAM and AIGC-based inpainting principles to enable flexible, context-aware shadow removal.

Experimental results

Research questions

• RQ1 Can SAM be effectively repurposed for zero-shot shadow detection without fine-tuning on shadow-specific data?
• RQ2 How does using SAM’s segmentation mask as a prior improve shadow detection and removal performance compared to traditional methods?
• RQ3 To what extent does a second-order deep unfolding network enhance shadow removal accuracy and detail preservation?
• RQ4 Can the proposed one-stage framework generalize to real-world images beyond benchmark datasets like ISTD and SRD?
• RQ5 How does the method compare to state-of-the-art shadow removal models in terms of quantitative metrics and visual quality?

Key findings

• On the ISTD dataset, SAM-helps-Shadow achieved a RMSE-all of 5.09, outperforming previous SOTA methods such as Bijetive (4.78) and SpA-Former (6.86), with a PSNR of 28.85 and SSIM of 0.959.
• On the SRD dataset, the method achieved a RMSE-all of 4.79, surpassing Bejetive (4.15) and SpA-Former (6.86), with a PSNR of 30.72 and SSIM of 0.952.
• The method demonstrated strong zero-shot generalization, effectively removing shadows in real-world images without requiring any fine-tuning on shadow-labeled data.
• Ablation studies confirmed that the proposed method outperformed both standard SAM and Image Inpainting-based approaches in segmentation accuracy and de-shadowing quality.
• The integration of SAM’s mask as a spatial prior significantly improved boundary consistency and reduced artifacts in shadow-removed images.
• The second-order deep unfolding network contributed to higher fidelity results, particularly in preserving texture and color consistency in shadowed regions.