[Paper Review] Structural Residual Learning for Single Image Rain Removal

Wang Hong, Yichen Wu+4 more|arXiv (Cornell University)|May 19, 2020

Image Enhancement Techniques55 references4 citations

TL;DR

This paper proposes a Structural Residual Network (SRNet) for single-image rain removal that enforces output residuals to conform to intrinsic rain streak structures, improving generalization to unseen rain patterns. By embedding prior knowledge of rain morphology into the network via multi-scale, dilated residual blocks and global residual learning, SRNet achieves state-of-the-art performance on both synthetic and real-world datasets with superior robustness and detail preservation.

ABSTRACT

To alleviate the adverse effect of rain streaks in image processing tasks, CNN-based single image rain removal methods have been recently proposed. However, the performance of these deep learning methods largely relies on the covering range of rain shapes contained in the pre-collected training rainy-clean image pairs. This makes them easily trapped into the overfitting-to-the-training-samples issue and cannot finely generalize to practical rainy images with complex and diverse rain streaks. Against this generalization issue, this study proposes a new network architecture by enforcing the output residual of the network possess intrinsic rain structures. Such a structural residual setting guarantees the rain layer extracted by the network finely comply with the prior knowledge of general rain streaks, and thus regulates sound rain shapes capable of being well extracted from rainy images in both training and predicting stages. Such a general regularization function naturally leads to both its better training accuracy and testing generalization capability even for those non-seen rain configurations. Such superiority is comprehensively substantiated by experiments implemented on synthetic and real datasets both visually and quantitatively as compared with current state-of-the-art methods.

Motivation & Objective

• To address the generalization failure of existing CNN-based deraining methods on unseen rain patterns not present in training data.
• To improve robustness and generalization in single-image rain removal by incorporating prior knowledge of rain streak structures into the network architecture.
• To develop a network that maintains high performance on real-world rainy images without requiring large-scale, diverse training pairs.
• To enable effective rain removal in video deraining using a single-image model, even without temporal information.
• To reduce overfitting to training samples by embedding structural inductive bias into the residual learning process.

Proposed method

• The proposed SRNet uses three parallel subnetworks with different dilated factors to capture multi-scale rain structures, mimicking the repetitive and directional nature of real rain streaks.
• A structural residual learning module is introduced, where the network's output residual is constrained to align with prior knowledge of rain patterns, such as consistent shape, thickness, and direction.
• The network employs an encoder-decoder architecture with skip connections and global residual learning to preserve feature fidelity and enhance training stability.
• Max-pooling and max-unpooling layers are used in the encoder-decoder path to maintain spatial resolution and enable precise feature reconstruction.
• Weight sharing across parallel branches reduces model parameters without sacrificing generalization, enabling a compact yet effective architecture.
• The method is trained end-to-end on synthetic rainy-clean image pairs, with no need for real-world paired data during training.

Experimental results

Research questions

• RQ1 Can enforcing structural priors on the residual output improve generalization in single-image deraining beyond training distribution?
• RQ2 How does multi-scale, dilated residual learning contribute to capturing diverse rain streak patterns not seen during training?
• RQ3 To what extent can a single-image deraining model outperform video-based deraining methods when applied to real rainy videos without temporal modeling?
• RQ4 Does structural residual learning lead to better preservation of background textures and fine details compared to standard residual networks?
• RQ5 Can weight sharing in parallel branches maintain performance while reducing model complexity and inference cost?

Key findings

• SRNet achieves a PSNR of 35.31 and SSIM of 0.9448 on the SPA-Data benchmark, outperforming all competing state-of-the-art methods in both quantitative and qualitative evaluations.
• Even without using temporal information, SRNet achieves competitive deraining performance on real rainy video frames, outperforming several video-based deraining methods in preserving image details like tree trunks and textures.
• The variant with max-unpooling (B_d) outperforms most other variants and competing methods on real deraining, demonstrating the effectiveness of the encoder-decoder design.
• The model with weight sharing (B_e) achieves comparable performance to the full model (B_f) while significantly reducing parameters, proving the efficiency of the architecture.
• Visualizations confirm that the extracted rain layer and sparse feature maps align well with real rain streak patterns, validating the structural inductive bias of the method.
• On real-world video frames with complex motion and low-light conditions, SRNet effectively removes rain streaks while minimizing artifacts, unlike model-based or deep learning-based video deraining methods that introduce distortions.