[Paper Review] Semi-Supervised Video Deraining with Dynamical Rain Generator

Zongsheng Yue, Jianwen Xie+2 more|arXiv (Cornell University)|Mar 14, 2021

Image Enhancement Techniques65 references4 citations

TL;DR

This paper proposes a semi-supervised video deraining method using a dynamical rain generator that models rain layers as a spatial-temporal process via emission and transition models parameterized by deep neural networks. By jointly leveraging labeled synthetic data and unlabeled real rainy videos, the method improves generalization to real-world conditions, achieving state-of-the-art performance on both synthetic and real benchmarks, especially in handling complex, diverse rain patterns.

ABSTRACT

While deep learning (DL)-based video deraining methods have achieved significant success recently, they still exist two major drawbacks. Firstly, most of them do not sufficiently model the characteristics of rain layers of rainy videos. In fact, the rain layers exhibit strong physical properties (e.g., direction, scale and thickness) in spatial dimension and natural continuities in temporal dimension, and thus can be generally modelled by the spatial-temporal process in statistics. Secondly, current DL-based methods seriously depend on the labeled synthetic training data, whose rain types are always deviated from those in unlabeled real data. Such gap between synthetic and real data sets leads to poor performance when applying them in real scenarios. Against these issues, this paper proposes a new semi-supervised video deraining method, in which a dynamic rain generator is employed to fit the rain layer, expecting to better depict its insightful characteristics. Specifically, such dynamic generator consists of one emission model and one transition model to simultaneously encode the spatially physical structure and temporally continuous changes of rain streaks, respectively, which both are parameterized as deep neural networks (DNNs). Further more, different prior formats are designed for the labeled synthetic and unlabeled real data, so as to fully exploit the common knowledge underlying them. Last but not least, we also design a Monte Carlo EM algorithm to solve this model. Extensive experiments are conducted to verify the superiorities of the proposed semi-supervised deraining model.

Motivation & Objective

• To address the gap between synthetic and real rainy video data, which limits the generalization of deep learning-based deraining methods.
• To model the intrinsic spatial and temporal characteristics of rain layers—such as direction, scale, thickness, velocity, and acceleration—using a statistical spatial-temporal process.
• To develop a semi-supervised learning framework that exploits shared knowledge between labeled synthetic data and unlabeled real rainy videos.
• To improve robustness and generalization in real-world deraining scenarios where synthetic data alone underperforms.

Proposed method

• A dynamical rain generator is introduced, consisting of an emission model for spatial appearance and a transition model for temporal dynamics of rain streaks, both parameterized by deep neural networks.
• The model uses a Monte Carlo-based Expectation-Maximization (EM) algorithm to learn parameters by optimizing a likelihood term and an MRF-based regularizer.
• A semi-supervised learning paradigm is adopted, where labeled synthetic data and unlabeled real rainy videos are jointly used to improve generalization.
• Different prior formats are designed for synthetic and real data to exploit their underlying common knowledge while accounting for distribution gaps.
• The likelihood term in the objective function models the probability of rain layers given the observed rainy video, while the MRF prior enforces spatial consistency across frames.
• The hyper-parameter ρ controls the trade-off between supervision from ground-truth labels and the MRF prior, with ρ = 0.5 selected for optimal balance.

Experimental results

Research questions

• RQ1 Can modeling rain layers as a spatial-temporal process improve video deraining performance compared to static or purely synthetic modeling?
• RQ2 How can unlabeled real rainy videos be effectively leveraged in a semi-supervised framework to improve generalization beyond synthetic data?
• RQ3 What is the impact of incorporating a dynamical rain generator with emission and transition models on handling diverse and complex real-world rain patterns?
• RQ4 How does the MRF prior contribute to preventing overfitting on synthetic data while enhancing performance on real data?

Key findings

• The proposed S2VD method achieves state-of-the-art performance on both synthetic and real-world benchmarks, with a 1.01 dB PSNR gain over MSE-only baseline on synthetic data.
• The addition of the likelihood term and MRF regularizer improves performance by 0.0071 SSIM and enhances generalization to real data, as shown in visual comparisons.
• S2VD generalizes well to real rainy videos not seen during training, as evidenced by superior visual results on the real testing set of NTURain.
• The method demonstrates robustness to diverse rain patterns, outperforming existing methods especially in complex real-world scenarios.
• The ablation study confirms that each component—likelihood term, MRF prior, and semi-supervised learning—contributes significantly to performance improvement.
• Failure cases occur under large camera motion or heavy rain streaks, indicating limitations in the current MRF prior's ability to model extreme dynamics.