[论文解读] Semi-supervised CNN for Single Image Rain Removal.

Wei Wei, Deyu Meng+2 more|arXiv (Cornell University)|Jul 29, 2018

Image Enhancement Techniques参考文献 26被引用 3

一句话总结

本文提出了一种半监督卷积神经网络用于单幅图像去雨，训练过程中同时利用了有标注的雨天-干净图像对以及无标注的雨天图像。通过将输入雨天图像与网络输出之间的残差建模为分块的高斯混合模型（Mixture of Gaussians, MoG），该方法结合了监督与无监督损失，减少了对稀缺的干净图像对的依赖，并提升了对未见雨型的泛化能力。

ABSTRACT

Single image rain removal is a typical inverse problem in computer vision. The deep learning technique has been verified to be effective to this task and achieved state-of-the-art performance. However, the method needs to pre-collect a large set of image pairs with/without rains for training, which not only makes the method laborsome to be practically implemented, but also tends to make the trained network bias to the training samples while less generalized to test samples with unseen rain types in training. To this issue, this paper firstly proposes a semisupervised learning paradigm to this task. Different from traditional deep learning methods which use only supervised image pairs with/without rains, we put the real rainy images, without need of their clean ones, into the network training process as well. This is realized by elaborately formulating the residual between an input rainy image and its expected network output (clear image without rains) as a concise patch-wised Mixture of Gaussians distribution. The entire objective function for training network is thus the combination of the supervised data loss (least square loss between input clear image and the network output) and the unsupervised data loss. In this way, all such unsupervised rainy images, which is much easier to collect than supervised ones, can be rationally fed into the network training process, and thus both the short-of-training-sample and bias-to-supervised-sample issues can be evidently alleviated. Experiments implemented on synthetic and real data experiments verify the superiority of our model as compared to the state-of-the-arts.

研究动机与目标

• 解决单幅图像去雨任务中配对雨天-干净图像数据集稀缺的问题，以训练深度学习模型。
• 通过在训练过程中引入无标注雨天图像，减少模型对监督训练数据中特定雨型的偏差。
• 通过利用大量无对应干净图像的真实雨天图像，提升模型对未见雨型模式的泛化能力。
• 开发一种结合监督与无监督学习的训练范式，以最小化人工标注数据的需求。

提出的方法

• 将输入雨天图像与网络预测的干净输出之间的残差建模为分块的高斯混合模型（MoG）分布。
• 提出一种基于MoG分布的无监督数据损失，用于在无真实干净图像作为监督信号的情况下训练无标注雨天图像。
• 将监督损失（预测输出与真实干净图像之间的最小二乘误差）与无监督损失相结合，实现网络的端到端训练。
• 利用MoG模型估计在给定输入雨天图像条件下，网络输出的似然性，从而实现来自无标注数据的自监督。
• 通过联合目标函数同时优化监督与无监督组件，训练CNN模型。
• 利用MoG的统计特性，指导网络在无配对干净标签的情况下学习干净图像的表征。

实验结果

研究问题

• RQ1 能否在无需对应干净图像的情况下，有效利用无标注雨天图像来训练去雨网络？
• RQ2 通过分块高斯混合模型实现无监督学习，是否能提升模型对未见雨型的泛化能力？
• RQ3 所提出的半监督方法在多大程度上降低了对大规模有标注雨天-干净图像对的依赖？
• RQ4 联合优化监督与无监督损失对合成数据集与真实世界去雨基准测试的性能有何影响？

主要发现

• 所提出的半监督方法在合成与真实雨天图像数据集上均优于当前最先进方法。
• 由于在训练过程中引入了多样化的无标注雨天图像，模型对未见雨型的泛化能力得到显著提升。
• 无监督损失组件能有效引导网络学习有意义的干净图像表征，即使在无真实干净图像的情况下亦然。
• 该方法显著减少了对大规模有标注雨天-干净图像对的需求，使其在真实世界部署中更具实用性。
• 实验表明，监督与无监督损失的联合训练可带来更鲁棒、更精确的去雨结果。
• 残差的高斯混合模型形式实现了有效的自监督，显著提升了模型在真实世界测试图像上的性能。