[Paper Review] Shadow Removal via Shadow Image Decomposition

Hieu Lê, Dimitris Samaras|arXiv (Cornell University)|Aug 23, 2019

Image Enhancement Techniques41 references4 citations

TL;DR

This paper proposes a deep learning framework for shadow removal that decomposes shadow images using a physical illumination model, estimating shadow parameters and matte via two networks (SP-Net and M-Net). By leveraging this decomposition, the method achieves state-of-the-art performance on the ISTD dataset, reducing shadow area RMSE from 13.3 to 7.4 through both model refinement and data augmentation.

ABSTRACT

We propose a novel deep learning method for shadow removal. Inspired by physical models of shadow formation, we use a linear illumination transformation to model the shadow effects in the image that allows the shadow image to be expressed as a combination of the shadow-free image, the shadow parameters, and a matte layer. We use two deep networks, namely SP-Net and M-Net, to predict the shadow parameters and the shadow matte respectively. This system allows us to remove the shadow effects on the images. We train and test our framework on the most challenging shadow removal dataset (ISTD). Compared to the state-of-the-art method, our model achieves a 40% error reduction in terms of root mean square error (RMSE) for the shadow area, reducing RMSE from 13.3 to 7.9. Moreover, we create an augmented ISTD dataset based on an image decomposition system by modifying the shadow parameters to generate new synthetic shadow images. Training our model on this new augmented ISTD dataset further lowers the RMSE on the shadow area to 7.4.

Motivation & Objective

• Address the challenge of realistic shadow removal in single images, where shadows degrade performance in computer vision tasks.
• Overcome limitations of end-to-end deep learning methods that produce blurry or artifact-ridden outputs by incorporating physical illumination models.
• Develop an automated, fully supervised method to estimate shadow parameters and matte without user intervention.
• Improve model generalization by generating synthetic shadow images through parameter manipulation for data augmentation.
• Achieve state-of-the-art performance on the challenging ISTD benchmark dataset for shadow removal.

Proposed method

• Model shadow formation using a linear illumination transformation per color channel: $I^\text{darkened}_i = (I^\text{shadow-free}_i - b) \cdot w^{-1}$, where $w$ and $b$ are shadow parameters.
• Use SP-Net, a deep neural network, to predict the shadow parameters $w$ and $b$ from input shadow images.
• Use M-Net, a second deep network, to predict the shadow matte $\alpha$, which controls the per-pixel blending between shadowed and relit regions.
• Reconstruct the shadow-free image via image decomposition: $I^\text{shadow-free} = I^\text{shadow} \cdot \alpha + I^\text{darkened} \cdot (1 - \alpha)$, where $I^\text{darkened}$ is relit using estimated $w$ and $b$.
• Generate synthetic shadow images by modifying the scaling factor $w$ via $w_{\text{syn}} = w \times k$ with $k \in \{0.8, 0.9, 1.1, 1.2\}$, creating augmented training data.
• Train the full SP+M-Net system on the original and augmented ISTD dataset to improve generalization and performance.

Experimental results

Research questions

• RQ1 Can a deep learning framework grounded in a simplified physical illumination model outperform end-to-end deep learning methods in shadow removal quality?
• RQ2 To what extent can shadow parameters and matte be accurately predicted in a fully automated, end-to-end manner using deep networks?
• RQ3 How effective is the proposed method for generating realistic synthetic shadow images for data augmentation?
• RQ4 Can data augmentation via parameterized shadow editing improve generalization and reduce error on benchmark datasets?
• RQ5 What is the impact of combining parameter estimation and matte prediction on shadow removal performance compared to using only one component?

Key findings

• The proposed SP+M-Net framework achieves a 40% reduction in root mean square error (RMSE) on the shadow area compared to the previous state-of-the-art, reducing RMSE from 13.3 to 7.9.
• Training on an augmented ISTD dataset—generated by modifying shadow parameters—further reduces RMSE on the shadow area to 7.4, a 6% improvement over the base model.
• SP-Net alone achieves a 29% improvement in RMSE on the shadow area (from 13.3 to 9.5), demonstrating the effectiveness of parameter estimation without matte prediction.
• The method produces high-quality, artifact-free results by avoiding direct per-pixel prediction, preserving image details and color fidelity.
• The proposed data augmentation strategy effectively expands the training set by four times (5320 synthetic images), significantly improving model robustness.
• Qualitative comparisons show that the method produces more realistic and consistent shadow removal results than prior state-of-the-art methods, including Guo et al., Yang et al., Gong et al., and Wang et al.