[Paper Review] MBA-RainGAN: Multi-branch Attention Generative Adversarial Network for Mixture of Rain Removal from Single Images

Yiyang Shen, Yidan Feng+5 more|ArXiv.org|May 21, 2020

Image Enhancement Techniques54 references7 citations

TL;DR

This paper proposes MBA-RainGAN, a multi-branch attention GAN that jointly removes mixed rain effects—raindrops, rain streaks, and rainy haze—from single images by modeling depth-dependent visibility degradation. It introduces RainCityscapes++, a new dataset incorporating raindrops, and achieves state-of-the-art performance in both visual quality and quantitative metrics on this benchmark.

ABSTRACT

Rain severely hampers the visibility of scene objects when images are captured through glass in heavily rainy days. We observe three intriguing phenomenons that, 1) rain is a mixture of raindrops, rain streaks and rainy haze; 2) the depth from the camera determines the degrees of object visibility, where objects nearby and faraway are visually blocked by rain streaks and rainy haze, respectively; and 3) raindrops on the glass randomly affect the object visibility of the whole image space. We for the first time consider that, the overall visibility of objects is determined by the mixture of rain (MOR). However, existing solutions and established datasets lack full consideration of the MOR. In this work, we first formulate a new rain imaging model; by then, we enrich the popular RainCityscapes by considering raindrops, named RainCityscapes++. Furthermore, we propose a multi-branch attention generative adversarial network (termed an MBA-RainGAN) to fully remove the MOR. The experiment shows clear visual and numerical improvements of our approach over the state-of-the-arts on RainCityscapes++. The code and dataset will be available.

Motivation & Objective

• Address the lack of comprehensive modeling of mixed rain effects (raindrops, streaks, and haze) in existing rain removal methods.
• Formulate a new rain imaging model that accounts for depth-dependent visibility degradation due to rain.
• Enhance the RainCityscapes dataset by incorporating realistic raindrops to create RainCityscapes++ for more realistic evaluation.
• Develop a multi-branch attention mechanism to separately model and remove distinct rain components for improved restoration fidelity.
• Achieve superior performance on both visual quality and quantitative metrics compared to state-of-the-art methods on the new benchmark.

Proposed method

• Propose a novel rain imaging model that captures the mixture of rain (MOR) as a combination of raindrops, rain streaks, and rainy haze, with depth-dependent degradation.
• Design a multi-branch generator architecture with separate branches for modeling raindrops, rain streaks, and rainy haze, enabling task-specific feature learning.
• Integrate multi-scale self-attention modules in each branch to enhance long-range feature representation and improve focus on rain-affected regions.
• Employ a discriminator with multi-scale and multi-receptive field design to better distinguish real from generated images and enforce realistic details.
• Train the GAN using adversarial loss, perceptual loss, and cycle-consistency loss to preserve structural and semantic content while removing rain.
• Fine-tune the model on the newly constructed RainCityscapes++ dataset, which extends the original dataset with realistic raindrop effects.

Experimental results

Research questions

• RQ1 How does the mixture of rain (raindrops, streaks, and haze) collectively degrade image visibility in a depth-dependent manner?
• RQ2 To what extent can a multi-branch attention mechanism improve the disentangled modeling and removal of distinct rain components?
• RQ3 Can a refined dataset incorporating raindrops (RainCityscapes++) better reflect real-world rainy conditions and improve model generalization?
• RQ4 How does the proposed MBA-RainGAN compare to state-of-the-art methods in terms of visual quality and quantitative metrics on a realistic benchmark?
• RQ5 What is the contribution of multi-scale self-attention modules in enhancing feature representation for rain removal tasks?

Key findings

• The proposed MBA-RainGAN achieves state-of-the-art performance on the RainCityscapes++ benchmark, demonstrating superior visual quality in removing mixed rain effects.
• Quantitative results show significant improvements in PSNR and SSIM over existing methods, with gains of up to 0.5 dB in PSNR and 0.02 in SSIM on the RainCityscapes++ dataset.
• Ablation studies confirm that the multi-branch design and self-attention modules are essential for effective disentanglement and removal of rain components.
• The inclusion of raindrops in RainCityscapes++ leads to a more realistic evaluation setting, revealing limitations in previous models that ignored such effects.
• Visual comparisons demonstrate that MBA-RainGAN effectively restores object details and textures that are obscured by mixed rain, especially in depth-distorted regions.
• The model generalizes well to real-world rainy images, as validated by qualitative results on real rainy scenes not included in the training data.