[Paper Review] Syn2Real Transfer Learning for Image Deraining using Gaussian Processes

Rajeev Yasarla, Vishwanath A. Sindagi+1 more|arXiv (Cornell University)|Jun 10, 2020

Image Enhancement Techniques32 references4 citations

TL;DR

This paper proposes a Gaussian Process-based semi-supervised learning framework that enables deep neural networks to leverage unlabeled real-world rainy images during training, improving generalization beyond synthetic data. By modeling latent features of labeled and unlabeled data jointly via GP to generate pseudo-ground-truth supervision, the method achieves performance comparable to fully supervised training even with only 60% labeled data, outperforming existing SOTA methods on real-world deraining benchmarks.

ABSTRACT

Recent CNN-based methods for image deraining have achieved excellent performance in terms of reconstruction error as well as visual quality. However, these methods are limited in the sense that they can be trained only on fully labeled data. Due to various challenges in obtaining real world fully-labeled image deraining datasets, existing methods are trained only on synthetically generated data and hence, generalize poorly to real-world images. The use of real-world data in training image deraining networks is relatively less explored in the literature. We propose a Gaussian Process-based semi-supervised learning framework which enables the network in learning to derain using synthetic dataset while generalizing better using unlabeled real-world images. Through extensive experiments and ablations on several challenging datasets (such as Rain800, Rain200H and DDN-SIRR), we show that the proposed method, when trained on limited labeled data, achieves on-par performance with fully-labeled training. Additionally, we demonstrate that using unlabeled real-world images in the proposed GP-based framework results in superior performance as compared to existing methods. Code is available at: https://github.com/rajeevyasarla/Syn2Real

Motivation & Objective

• To address the domain shift between synthetic and real-world rainy images that limits performance of existing CNN-based deraining models.
• To enable effective use of unlabeled real-world images in training, overcoming the scarcity of real-world labeled data.
• To develop a robust semi-supervised learning framework that improves generalization without relying on parametric assumptions like GMMs.
• To demonstrate that limited labeled data combined with unlabeled real-world data can achieve performance on par with fully supervised training.

Proposed method

• The method uses a deep CNN with supervised loss (L1 and perceptual loss) on labeled synthetic rainy images.
• For unlabeled real-world images, it estimates pseudo-ground-truth in the latent space using a Gaussian Process that models both labeled and unlabeled latent features jointly.
• The GP is trained iteratively, refining pseudo-labels for unlabeled data based on the evolving feature representations.
• The framework alternates between optimizing the network on labeled data and updating the GP prior using both labeled and unlabeled latent vectors.
• It avoids parametric assumptions by using a non-parametric GP, eliminating the need to predefine mixture components as in GMM-based methods.
• The approach is validated through ablation studies and comparisons on Rain800, Rain200H, and DDN-SIRR datasets, showing consistent gains with unlabeled data.

Experimental results

Research questions

• RQ1 Can a semi-supervised learning framework effectively improve image deraining generalization using only unlabeled real-world rainy images?
• RQ2 How does a GP-based pseudo-labeling strategy compare to GMM-based methods in modeling complex, multi-modal rain residuals?
• RQ3 To what extent can performance be maintained or improved when using only 60% of the labeled data by incorporating unlabeled real-world data?
• RQ4 Does the proposed method reduce the domain shift between synthetic and real rainy images more effectively than existing SOTA approaches?
• RQ5 How sensitive is the method to hyperparameters such as GP kernel choice and regularization?

Key findings

• Using only 60% labeled data (with unlabeled data) achieves PSNR of 23.70 dB on Rain800, matching the 23.74 dB performance of full supervised training.
• On Rain200H, the method achieves 25.26 dB PSNR with 60% labeled data and unlabeled data, compared to 25.27 dB with 100% labeled data.
• The proposed method reduces PSNR drop from 21.31 dB (10% labeled only) to 22.02 dB when unlabeled data is used, showing consistent gains across data splits.
• The method outperforms SIRR [49] in both PSNR and SSIM on real-world datasets, demonstrating superior generalization.
• Ablation studies confirm that the GP-based pseudo-labeling strategy is more robust than GMM-based approaches, especially in early training stages.
• The method achieves state-of-the-art performance on the DDN-SIRR real-world benchmark, validating its effectiveness in real-world deployment.