[论文解读] From Rain Removal to Rain Generation

Hong Wang, Zongsheng Yue+3 more|arXiv (Cornell University)|Aug 8, 2020

Image Enhancement Techniques参考文献 2被引用 5

一句话总结

本文提出了一种全贝叶斯生成模型，通过使用CNN参数化的变分推断框架，学习雨痕的内在物理因素（如方向、尺度和厚度）。通过将雨层建模为隐式生成器，该方法能够自动、多样化地生成雨斑图像以增强训练数据，显著提升单图去雨性能，同时减少对人工收集的成对数据集的依赖。

ABSTRACT

Single image deraining is an important yet challenging issue due to the complex and diverse rain structures in real scenes. Currently, the state-of-the-art performance on this task is achieved by deep learning (DL)-based methods that mainly benefit from abundant pre-collected paired rainy-clean samples either manually synthesized or semi-automatically generated under human supervision. This tends to bring a large labor for data collection and more importantly, such manner neglects to elaborately explore the intrinsic generative mechanism of rain streaks which should be related to the most insightful understanding of the task. Against this issue, we investigate the generative process of rainy image and construct a full Bayesian generative model for generating rains from automatically extracted latent variables that represent physical structural factors for depicting rains, like direction, scale, and thickness. To solve this model, we propose an algorithm where the posteriors of latent variables are parameterized as CNNs and all the involved parameters can be inferred under a concise variational inference framework in a data-driven manner. Especially, the rain layer is modeled as an implicit distribution, parameterized as a generator, which avoids subjective prior assumptions on rains as in traditional model-based methods. More practically, from the learned generator, rain patches can be automatically generated and utilized to simulate diverse training pairs so as to enrich and augment the existing benchmark datasets. Comprehensive experiments substantiate that the proposed model has fine capability of generating plausible samples that not only helps significantly improve the deraining performance of current DL-based single image derainers, but also largely loosens the requirement of large training sample pre-collection for the task.

研究动机与目标

• 为解决当前基于深度学习的去雨方法严重依赖大规模人工收集的成对雨天-清晰图像数据集的问题。
• 通过建模方向、尺度和厚度等物理结构因素，揭示雨痕的内在生成机制。
• 开发一种数据驱动的端到端框架，无需对雨痕模式施加主观先验假设，即可生成逼真的雨斑。
• 通过利用生成的雨斑增强现有基准数据集，提升去雨模型的性能。

提出的方法

• 构建全贝叶斯生成模型，利用编码物理属性（如方向、尺度和厚度）的潜在变量来表示雨痕。
• 使用卷积神经网络（CNNs）参数化这些潜在变量的后验分布，以在变分推断框架内实现高效推断。
• 通过生成器网络将雨层建模为隐式分布，避免了传统基于模型方法中常见的显式参数假设。
• 该模型支持端到端训练，所有参数均通过变分推断以数据驱动方式推断。
• 从训练好的生成器中合成生成的雨斑，并用于创建多样化、逼真的去雨模型训练对。
• 该方法支持无需人工标注监督的自动数据增强，可生成新的、合理的雨天图像。

实验结果

研究问题

• RQ1 如何建模雨痕的内在物理生成过程，以实现逼真的雨斑合成？
• RQ2 一种基于数据驱动的贝叶斯框架，结合CNN参数化的后验分布，能否在不依赖手工设计先验的情况下有效推断潜在雨因子？
• RQ3 通过自动生成的雨斑能在多大程度上提升现有基于深度学习的去雨模型的性能？
• RQ4 所提方法在多大程度上减少了对大规模、人工收集的成对雨天-清晰图像数据集的依赖？
• RQ5 使用合成雨数据对去雨网络的泛化能力和鲁棒性有何影响？

主要发现

• 所提模型生成的雨斑高度逼真，其结构和多样性与真实世界雨痕极为相似。
• 生成的雨斑显著提升了最先进深度学习模型在基准数据集上的去雨性能。
• 该方法通过合成雨生成实现有效的数据增强，显著减少了对大规模人工收集成对训练数据的需求。
• 隐式生成器模型优于依赖固定参数假设雨模式的传统方法。
• 变分推断框架支持高效且可扩展的端到端可微训练，有利于实际部署。
• 实证结果表明，使用增强数据训练的模型在性能上优于仅使用原始数据集训练的模型。