[Paper Review] Image De-raining Using a Conditional Generative Adversarial Network

He Zhang, Vishwanath A. Sindagi+1 more|arXiv (Cornell University)|Jan 21, 2017

Image Enhancement Techniques80 references249 citations

TL;DR

The paper introduces ID-CGAN, a conditional GAN with a densely connected generator and a multi-scale discriminator to perform single-image de-raining, achieving superior visual/quantitative results and improving downstream object detection.

ABSTRACT

Severe weather conditions such as rain and snow adversely affect the visual quality of images captured under such conditions thus rendering them useless for further usage and sharing. In addition, such degraded images drastically affect performance of vision systems. Hence, it is important to solve the problem of single image de-raining/de-snowing. However, this is a difficult problem to solve due to its inherent ill-posed nature. Existing approaches attempt to introduce prior information to convert it into a well-posed problem. In this paper, we investigate a new point of view in addressing the single image de-raining problem. Instead of focusing only on deciding what is a good prior or a good framework to achieve good quantitative and qualitative performance, we also ensure that the de-rained image itself does not degrade the performance of a given computer vision algorithm such as detection and classification. In other words, the de-rained result should be indistinguishable from its corresponding clear image to a given discriminator. This criterion can be directly incorporated into the optimization framework by using the recently introduced conditional generative adversarial networks (GANs). To minimize artifacts introduced by GANs and ensure better visual quality, a new refined loss function is introduced. Based on this, we propose a novel single image de-raining method called Image De-raining Conditional General Adversarial Network (ID-CGAN), which considers quantitative, visual and also discriminative performance into the objective function. Experiments evaluated on synthetic images and real images show that the proposed method outperforms many recent state-of-the-art single image de-raining methods in terms of quantitative and visual performance.

Motivation & Objective

• Motivate the problem of rain-induced image degradation and its impact on vision systems.
• Address the ill-posed nature of single image de-raining by incorporating discriminative and perceptual criteria into the optimization.
• Propose a CGAN-based framework (ID-CGAN) tailored for rain removal without post-processing.
• Design a densely connected generator and a multi-scale discriminator to capture global and local context during deraining.
• Demonstrate effectiveness on synthetic and real-world datasets and show improved object detection results.

Proposed method

• Use a conditional GAN to learn a mapping from a rainy image x to a de-rained image y, conditioned on x.
• Introduce a densely-connected generator with skip connections to preserve details during rain removal.
• Employ a multi-scale discriminator to leverage both local and global information for real/fake discrimination.
• Define a refined perceptual loss L_RP that combines pixel-level loss L_E, adversarial loss L_A, and perceptual loss L_P with tuned weights.
• Combine Euclidean, perceptual, and adversarial losses to guide training and reduce GAN-induced artifacts (L_RP = L_E + lambda_a L_A + lambda_p L_P).
• Train with synthetic and real rain datasets; evaluate using PSNR, SSIM, UQI, VIF, and assess downstream object detection performance with Faster-RCNN.

Experimental results

Research questions

• RQ1 Can a conditional GAN framework effectively separate rain streaks from the clean background in a single image?
• RQ2 Does a densely connected generator plus a multi-scale discriminator improve de-raining quality over baselines?
• RQ3 Does incorporating a refined perceptual loss reduce artifacts and improve both visual and quantitative metrics?
• RQ4 Is the proposed method beneficial for downstream tasks such as object detection on rain-degraded imagery?

Key findings

• ID-CGAN outperforms several state-of-the-art single image de-raining methods in visual quality and quantitative metrics on synthetic data.
• The multi-scale discriminator helps recover fine texture details that single-scale discriminators miss.
• The refined perceptual loss reduces artifacts and improves sharpness and detail preservation compared to using only pixel or adversarial losses.
• ID-CGAN improves detection performance in object detection pipelines (e.g., Faster-RCNN) on rain-degraded images.
• Ablation studies show the combination of Euclidean, perceptual, and adversarial losses with a multi-scale discriminator yields the best results among evaluated configurations.