[Paper Review] SAPNet: Segmentation-Aware Progressive Network for Perceptual Contrastive Deraining

Shen Zheng, Changjie Lu+2 more|arXiv (Cornell University)|Nov 16, 2021

Image Enhancement Techniques4 citations

TL;DR

SAPNet proposes a segmentation-aware progressive network for single-image deraining that combines a lightweight progressive dilated unit (PDU) for multi-scale rain streak modeling, unsupervised background segmentation (UBS) to preserve semantic details, and a perceptual contrastive loss (PCL) with learned perceptual similarity to align derained outputs with ground-truth in both pixel-wise and perceptual space. The method achieves state-of-the-art performance on synthetic and real-world deraining benchmarks, significantly improving object detection and semantic segmentation accuracy.

ABSTRACT

Deep learning algorithms have recently achieved promising deraining performances on both the natural and synthetic rainy datasets. As an essential low-level pre-processing stage, a deraining network should clear the rain streaks and preserve the fine semantic details. However, most existing methods only consider low-level image restoration. That limits their performances at high-level tasks requiring precise semantic information. To address this issue, in this paper, we present a segmentation-aware progressive network (SAPNet) based upon contrastive learning for single image deraining. We start our method with a lightweight derain network formed with progressive dilated units (PDU). The PDU can significantly expand the receptive field and characterize multi-scale rain streaks without the heavy computation on multi-scale images. A fundamental aspect of this work is an unsupervised background segmentation (UBS) network initialized with ImageNet and Gaussian weights. The UBS can faithfully preserve an image's semantic information and improve the generalization ability to unseen photos. Furthermore, we introduce a perceptual contrastive loss (PCL) and a learned perceptual image similarity loss (LPISL) to regulate model learning. By exploiting the rainy image and groundtruth as the negative and the positive sample in the VGG-16 latent space, we bridge the fine semantic details between the derained image and the groundtruth in a fully constrained manner. Comprehensive experiments on synthetic and real-world rainy images show our model surpasses top-performing methods and aids object detection and semantic segmentation with considerable efficacy. A Pytorch Implementation is available at https://github.com/ShenZheng2000/SAPNet-for-image-deraining.

Motivation & Objective

• Address the gap between low-level image restoration and high-level vision tasks by improving deraining models for semantic preservation.
• Overcome limitations of existing deraining methods that focus only on image quality metrics (e.g., PSNR, SSIM) without benefiting downstream tasks like object detection and segmentation.
• Reduce reliance on expensive annotated data by introducing unsupervised background segmentation to preserve semantic structure without segmentation labels.
• Enhance generalization to real-world rainy images with complex illumination and dense rain streaks through multi-scale contextual modeling and contrastive learning.
• Achieve real-time inference efficiency suitable for mobile and real-time applications like autonomous driving while maintaining high performance.

Proposed method

• Propose a progressive dilated unit (PDU) that expands the receptive field through dilated convolutions in a progressive, parameter-efficient manner, enabling effective multi-scale rain streak capture without heavy computation.
• Introduce an unsupervised background segmentation (UBS) network initialized with ImageNet and Gaussian weights to preserve semantic details during deraining, without requiring segmentation annotations.
• Design a perceptual contrastive loss (PCL) that treats the rainy image as a negative sample and the ground-truth as a positive sample in the VGG-16 feature space, encouraging the derained output to match the ground-truth in perceptual space.
• Combine PCL with a learned perceptual image similarity loss (LPISL) to jointly optimize for both pixel-level fidelity and perceptual realism.
• Integrate the derain network, UBS network, and VGG-16 backbone into a unified framework where features are aligned via contrastive learning and perceptual supervision.
• Train the model end-to-end using supervised loss on the deraining task, with PCL and LPISL providing additional perceptual regularization.

Figure 2: User study score $\uparrow$ and inference time $\downarrow$ comparison. The user study score (1-5) is averaged from real-rainy datasets including Rain800, SIRR and MOSS. The average run time (inference time) is calculated on images with size 512 $\times$ 512 with a single NVIDIA RTX 2080 T

Experimental results

Research questions

• RQ1 Can unsupervised background segmentation improve semantic preservation in deraining without requiring segmentation annotations?
• RQ2 To what extent does perceptual contrastive learning enhance the quality of derained images beyond standard pixel-wise losses?
• RQ3 How does the progressive dilated unit (PDU) improve multi-scale rain streak modeling compared to standard or multi-scale input approaches?
• RQ4 Does a deraining model optimized for perceptual similarity and contrastive alignment lead to measurable improvements in downstream high-level vision tasks like object detection and semantic segmentation?
• RQ5 Can the proposed method achieve state-of-the-art performance on both synthetic and real-world rainy datasets while maintaining low inference time and model size?

Key findings

• On the Rain100H benchmark, SAPNet achieves the highest PSNR (29.45) and SSIM (0.903), outperforming all prior state-of-the-art methods in both metrics.
• On the real-world SIRR dataset, SAPNet achieves the best BRISQUE score (17.54), indicating superior image quality and reduced distortion.
• On the MOSS dataset, SAPNet achieves the best UNIQUE score (1.05), demonstrating strong generalization to real-world rainy images with diverse weather and lighting conditions.
• In object detection on COCO150, SAPNet enables YOLOv3 to detect traffic lights and other small objects that are missed by competing models like DDN, Syn2Real, and EffDerain.
• In semantic segmentation on CityScape150, SAPNet produces the most accurate segmentation maps, closely matching the ground truth, while models like DDN and RESCAN fail to detect entire objects such as the left car.
• The ablation study confirms that the perceptual contrastive loss (PCL) significantly improves performance, while L1 contrastive loss degrades results, demonstrating the necessity of perceptual-level alignment.

Figure 3: Model architecture for SAPNet. SAPNet joins a derain network for supervised rain removal, a segmentation network for unsupervised background segmentation, and a VGG-16 network for perceptual contrast. The rainy image first enters the derain network for rain removal, using the groundtruth a