[Paper Review] Semi-supervised CNN for Single Image Rain Removal.

Wei Wei, Deyu Meng+2 more|arXiv (Cornell University)|Jul 29, 2018

Image Enhancement Techniques26 references3 citations

TL;DR

This paper proposes a semi-supervised CNN for single image rain removal that leverages both labeled rainy-clean image pairs and unlabeled rainy images during training. By modeling the residual between input rainy images and network outputs as a patch-wise Mixture of Gaussians, the method combines supervised and unsupervised losses, reducing reliance on scarce clean image pairs and improving generalization to unseen rain types.

ABSTRACT

Single image rain removal is a typical inverse problem in computer vision. The deep learning technique has been verified to be effective to this task and achieved state-of-the-art performance. However, the method needs to pre-collect a large set of image pairs with/without rains for training, which not only makes the method laborsome to be practically implemented, but also tends to make the trained network bias to the training samples while less generalized to test samples with unseen rain types in training. To this issue, this paper firstly proposes a semisupervised learning paradigm to this task. Different from traditional deep learning methods which use only supervised image pairs with/without rains, we put the real rainy images, without need of their clean ones, into the network training process as well. This is realized by elaborately formulating the residual between an input rainy image and its expected network output (clear image without rains) as a concise patch-wised Mixture of Gaussians distribution. The entire objective function for training network is thus the combination of the supervised data loss (least square loss between input clear image and the network output) and the unsupervised data loss. In this way, all such unsupervised rainy images, which is much easier to collect than supervised ones, can be rationally fed into the network training process, and thus both the short-of-training-sample and bias-to-supervised-sample issues can be evidently alleviated. Experiments implemented on synthetic and real data experiments verify the superiority of our model as compared to the state-of-the-arts.

Motivation & Objective

• Address the challenge of limited availability of paired rainy-clean image datasets for training deep learning models in single image rain removal.
• Reduce model bias toward specific rain types present in supervised training data by incorporating unlabeled rainy images during training.
• Improve generalization to unseen rain patterns by leveraging the abundance of real rainy images without requiring corresponding clean counterparts.
• Develop a training paradigm that combines supervised and unsupervised learning to enhance performance with minimal human-annotated data.

Proposed method

• Formulate the residual between an input rainy image and the network’s predicted clean output as a patch-wise Mixture of Gaussians (MoG) distribution.
• Introduce an unsupervised data loss based on the MoG distribution to train on unlabeled rainy images without ground-truth clean images.
• Combine the supervised loss (least squares between predicted and ground-truth clean images) with the unsupervised loss to train the network end-to-end.
• Use the MoG model to estimate the likelihood of the network’s output given the input rainy image, enabling self-supervision from unlabeled data.
• Train the CNN using a joint objective function that optimizes both supervised and unsupervised components simultaneously.
• Leverage the statistical properties of the MoG to guide the network in learning clean image representations even without paired clean labels.

Experimental results

Research questions

• RQ1 Can unlabeled rainy images be effectively utilized in training a rain removal network without requiring corresponding clean images?
• RQ2 Does incorporating unsupervised learning via a patch-wise Mixture of Gaussians improve model generalization to unseen rain types?
• RQ3 To what extent does the proposed semi-supervised approach reduce dependency on large-scale labeled rainy-clean image pairs?
• RQ4 How does the joint optimization of supervised and unsupervised losses affect performance on both synthetic and real-world rain removal benchmarks?

Key findings

• The proposed semi-supervised method achieves superior performance compared to state-of-the-art methods on both synthetic and real rainy image datasets.
• The model demonstrates improved generalization to unseen rain types due to the inclusion of diverse unlabeled rainy images during training.
• The unsupervised loss component effectively guides the network to learn meaningful clean image representations even without ground-truth clean images.
• The method reduces the need for large-scale labeled rainy-clean image pairs, making it more practical for real-world deployment.
• Experiments show that the joint training with supervised and unsupervised losses leads to more robust and accurate rain removal results.
• The Mixture of Gaussians formulation for residuals enables effective self-supervision, enhancing model performance on real-world test images.