[论文解读] Closing the Loop: Joint Rain Generation and Removal via Disentangled Image Translation

Yuntong Ye, Yi Chang+2 more|arXiv (Cornell University)|Mar 25, 2021

Image Enhancement Techniques参考文献 41被引用 6

一句话总结

本文提出一种基于解耦图像翻译的联合雨滴生成与去除框架，以弥合合成雨与真实雨之间的域差距。通过将有雨图像分解为干净背景与雨滴图层，该方法实现了具备循环一致性与对抗性损失的双向翻译，显著提升了对真实世界雨滴图像的泛化能力，同时保持了身份一致性并生成自然结果。

ABSTRACT

Existing deep learning-based image deraining methods have achieved promising performance for synthetic rainy images, typically rely on the pairs of sharp images and simulated rainy counterparts. However, these methods suffer from significant performance drop when facing the real rain, because of the huge gap between the simplified synthetic rain and the complex real rain. In this work, we argue that the rain generation and removal are the two sides of the same coin and should be tightly coupled. To close the loop, we propose to jointly learn real rain generation and removal procedure within a unified disentangled image translation framework. Specifically, we propose a bidirectional disentangled translation network, in which each unidirectional network contains two loops of joint rain generation and removal for both the real and synthetic rain image, respectively. Meanwhile, we enforce the disentanglement strategy by decomposing the rainy image into a clean background and rain layer (rain removal), in order to better preserve the identity background via both the cycle-consistency loss and adversarial loss, and ease the rain layer translating between the real and synthetic rainy image. A counterpart composition with the entanglement strategy is symmetrically applied for rain generation. Extensive experiments on synthetic and real-world rain datasets show the superiority of proposed method compared to state-of-the-arts.

研究动机与目标

• 解决基于深度学习的去雨方法中合成雨与真实雨之间的域差距问题。
• 通过联合学习雨滴生成与去除任务，提升对真实世界雨天图像的泛化能力。
• 通过将图像解耦为干净图层与雨滴图层，实现去雨过程中的背景身份保持。
• 通过从数据中隐式学习物理退化过程，弥合合成雨与真实雨之间的差距。
• 通过生成与去除任务的相互优化，提升真实雨天图像的去雨性能。

提出的方法

• 提出一种双向解耦翻译网络，包含两个单向分支，分别用于真实雨与合成雨的翻译。
• 将有雨图像分解为干净背景与雨滴图层，从而将身份保持与雨滴翻译解耦。
• 应用循环一致性损失与对抗性损失，以确保背景身份一致性和真实感翻译。
• 采用两阶段训练：先进行去雨，再进行雨滴生成，形成闭环训练流程。
• 在两个方向上对称地应用解耦机制——从有雨图像到干净图像的去雨，以及从干净图像到有雨图像的雨滴生成。
• 通过 t-SNE 的特征对齐验证了不同域之间雨滴图层翻译的有效性。

Figure 1: Illustration of rain generation and removal. (a) The hand-crafted simplified rain generation model. (b) Although the simulated rain has been well removed, the huge gap between synthetic training images and real-world testing images leads a significant performance drop in existing learning-

实验结果

研究问题

• RQ1 联合学习雨滴生成与去除是否能提升对真实世界雨天图像的泛化能力？
• RQ2 将背景与雨滴图层解耦是否能增强身份保持与翻译质量？
• RQ3 具有双向翻译的统一框架能否有效减小合成雨与真实雨之间的域差距？
• RQ4 与 CycleGAN 相比，所提方法在背景保持与真实雨滴生成方面表现如何？
• RQ5 从真实数据中端到端学习物理退化过程，在多大程度上提升了去雨性能？

主要发现

• 所提方法在合成雨与真实世界雨数据集上均达到当前最优性能，优于现有 SOTA 方法。
• 模型在两个方向上均能成功重建输入图像，展现出强大的循环一致性与身份保持能力。
• t-SNE 可视化结果证实，翻译后合成雨与真实雨滴图层的特征实现了有效对齐。
• 与 CycleGAN 相比，JRGR 通过聚焦于雨滴图层的翻译，同时保持背景不变，生成了更自然的结果。
• 当训练与测试背景相似时，该方法表现最佳，表明其在处理域内大范围变化方面存在局限性。
• 当训练与测试数据背景特征显著不同时（如城市与森林），方法出现失败，凸显了构建全面数据集的必要性。

Figure 2: Overview of the proposed joint rain generation and removal (JRGR) framework. The proposed method consists of synthetic to real (S2R) and real to synthetic (R2S) modules, aiming to bridge the gap between synthetic and real rainy images. Instead of the directly translating, we embed the rain