[Paper Review] Semantic Guided Single Image Reflection Removal

Yunfei Liu, Yu Li+2 more|arXiv (Cornell University)|Jul 27, 2019

Image Enhancement Techniques30 references7 citations

TL;DR

This paper proposes a multi-task deep learning framework, Semantic Guided Reflection Removal Network (SGR2N), that jointly performs single image reflection removal and semantic segmentation by leveraging high-level object semantics as guidance. By fusing semantic cues with reflection separation, the method achieves state-of-the-art performance in both tasks, significantly improving background clarity and semantic accuracy under challenging reflection conditions.

ABSTRACT

Reflection is common in images capturing scenes behind a glass window, which is not only a disturbance visually but also influence the performance of other computer vision algorithms. Single image reflection removal is an ill-posed problem because the color at each pixel needs to be separated into two values, i.e., the desired clear background and the reflection. To solve it, existing methods propose priors such as smoothness, color consistency. However, the low-level priors are not reliable in complex scenes, for instance, when capturing a real outdoor scene through a window, both the foreground and background contain both smooth and sharp area and a variety of color. In this paper, inspired by the fact that human can separate the two layers easily by recognizing the objects, we use the object semantic as guidance to force the same semantic object belong to the same layer. Extensive experiments on different datasets show that adding the semantic information offers a significant improvement to reflection separation. We also demonstrate the applications of the proposed method to other computer vision tasks.

Motivation & Objective

• To address the ill-posed nature of single-image reflection removal by incorporating high-level semantic information instead of relying solely on low-level priors.
• To improve reflection removal performance by jointly training for semantic segmentation and background/reflection separation.
• To demonstrate that semantic understanding enhances the accuracy and robustness of reflection removal in real-world scenarios.
• To validate the benefit of reflection removal as a preprocessing step for downstream vision tasks like object detection.

Proposed method

• A multi-task end-to-end deep learning framework is designed to simultaneously optimize for reflection removal and semantic segmentation.
• A semantic guidance block explicitly fuses semantic feature maps with feature representations from the reflection removal network to guide layer separation.
• The network uses an encoder-decoder architecture with skip connections to preserve spatial details for both tasks.
• The method leverages the assumption that background intensity is stronger than reflection, making background semantics more reliable for guidance.
• Loss functions include standard reconstruction loss for reflection removal and cross-entropy loss for semantic segmentation, jointly optimized during training.
• The model is trained on datasets with real reflection images and annotated semantic masks to enable joint optimization.

Experimental results

Research questions

• RQ1 Can high-level semantic cues improve the accuracy and robustness of single-image reflection removal compared to low-level priors?
• RQ2 How does joint learning of semantic segmentation and reflection removal enhance performance in both tasks?
• RQ3 To what extent does reflection removal improve downstream computer vision tasks like object detection?
• RQ4 How does the method perform under varying reflection intensities and complex scene conditions?

Key findings

• The proposed SGR2N achieves state-of-the-art performance on three benchmark datasets in both reflection removal (PSNR) and semantic segmentation (mIoU).
• On images with varying reflection intensities, SGR2N maintains high performance, with mIoU dropping only slightly when reflection strength is below 0.5.
• When reflection strength exceeds 0.5, semantic accuracy degrades, but SGR2N still outperforms baseline methods like ZN18 and IBCLN.
• In object detection experiments using YOLO-v3, mAP on reflection-contaminated images dropped by 12.42% compared to clean backgrounds, but improved significantly after reflection removal.
• The detection mAP on the reflection-removed output from SGR2N reached 42.83%, outperforming other methods and approaching the clean baseline.
• Visual results show that SGR2N produces clearer background and more distinct reflection layers, especially in complex scenes with strong reflections.