[论文解读] Image De-raining Using a Conditional Generative Adversarial Network

He Zhang, Vishwanath A. Sindagi+1 more|arXiv (Cornell University)|Jan 21, 2017

Image Enhancement Techniques参考文献 80被引用 249

一句话总结

该论文提出 ID-CGAN，一种带有密集连接生成器和多尺度判别器的条件 GAN，用于单图像去雾，取得优越的视觉/定量结果并提升下游目标检测。

ABSTRACT

Severe weather conditions such as rain and snow adversely affect the visual quality of images captured under such conditions thus rendering them useless for further usage and sharing. In addition, such degraded images drastically affect performance of vision systems. Hence, it is important to solve the problem of single image de-raining/de-snowing. However, this is a difficult problem to solve due to its inherent ill-posed nature. Existing approaches attempt to introduce prior information to convert it into a well-posed problem. In this paper, we investigate a new point of view in addressing the single image de-raining problem. Instead of focusing only on deciding what is a good prior or a good framework to achieve good quantitative and qualitative performance, we also ensure that the de-rained image itself does not degrade the performance of a given computer vision algorithm such as detection and classification. In other words, the de-rained result should be indistinguishable from its corresponding clear image to a given discriminator. This criterion can be directly incorporated into the optimization framework by using the recently introduced conditional generative adversarial networks (GANs). To minimize artifacts introduced by GANs and ensure better visual quality, a new refined loss function is introduced. Based on this, we propose a novel single image de-raining method called Image De-raining Conditional General Adversarial Network (ID-CGAN), which considers quantitative, visual and also discriminative performance into the objective function. Experiments evaluated on synthetic images and real images show that the proposed method outperforms many recent state-of-the-art single image de-raining methods in terms of quantitative and visual performance.

研究动机与目标

• 动机：雨滴引起的图像降质及其对视觉系统的影响。
• 通过将判别性和感知准则融入优化，解决单幅图像去雾/去雨的病态性。
• 提出一个基于 CGAN 的框架（ID-CGAN），专为无需后处理的去雨而设计。
• 设计一个密集连接的生成器和一个多尺度判别器，在去雨过程中捕捉全局和局部上下文。
• 在合成数据集和真实数据集上证明有效性，并显示对象检测结果的提升。

提出的方法

• 使用条件GAN学习从雨天图像 x 映射到去雨图像 y 的映射，条件依赖于 x。
• 引入带跳跃连接的密集连接生成器，以在去雨过程中保留细节。
• 采用多尺度判别器，利用局部和全局信息进行真/假判别。
• 定义一个 refined perceptual loss L_RP，结合像素级损失 L_E、对抗损失 L_A 和感知损失 L_P，并给出调整后的权重。
• 将欧几里得损失、感知损失和对抗损失结合起来以引导训练并减少GAN引起的伪影（L_RP = L_E + lambda_a L_A + lambda_p L_P）。
• 使用合成雨和真实雨数据集进行训练；用 PSNR、SSIM、UQI、VIF 进行评估，并使用 Faster-RCNN 评估下游目标检测性能。

实验结果

研究问题

• RQ1 条件 GAN 框架是否能够在单幅图像中有效地将雨滴条纹与清晰背景分离？
• RQ2 密集连接的生成器加上多尺度判别器是否在去雨质量上优于基线？
• RQ3 结合 refined perceptual loss 是否能减少伪影并同时提升视觉和定量指标？
• RQ4 所提出的方法对雨降图像的下游任务（如目标检测）是否有益？

主要发现

• ID-CGAN 在合成数据上在视觉质量和定量指标方面优于多种最新的单图像去雨方法。
• 多尺度判别器有助于恢复单尺度判别器遗漏的细粒度纹理细节。
• 相较于仅使用像素损失或对抗损失， refined perceptual loss 能减少伪影、提高锐利度并保留细节。
• ID-CGAN 提升了在雨降图像上的目标检测管线（如 Faster-RCNN）的检测性能。
• 消融研究表明，将欧氏、感知和对抗损失与多尺度判别器相结合，在所评估的配置中获得最佳结果。