[论文解读] Joint Rain Detection and Removal via Iterative Region Dependent Multi-Task Learning.

Wenhan Yang, Robby T. Tan+4 more|arXiv (Cornell University)|Sep 25, 2016

Image Enhancement Techniques参考文献 35被引用 42

一句话总结

本文提出了一种新颖的多任务深度学习框架，结合迭代反馈机制，用于从单幅图像中联合检测并去除雨水，尤其在强降雨条件下表现优异。通过使用二值图和空洞卷积分别建模雨痕与雨滴累积，结合迭代信息反馈网络，该方法在去除密集重叠的雨痕及类似薄雾的累积效应方面达到了最先进性能。

ABSTRACT

In this paper, we address a rain removal problem from a single image, even in the presence of heavy rain and rain accumulation. Our core ideas lie in our new rain image models and a novel deep learning architecture. We first modify the commonly used model, which is a linear combination of a rain streak layer and a background layer, by adding a binary map that locates rain streak regions. Second, we create a model consisting of a component representing rain accumulation (where individual streaks cannot be seen, and thus visually similar to mist or fog), and another component representing various shapes and directions of overlapping rain streaks, which normally happen in heavy rain. Based on the first model, we develop a multi-task deep learning architecture that learns the binary rain streak map, the appearance of rain streaks, and the clean background, which is our ultimate output. The additional binary map is critically beneficial, since its loss function can provide additional strong information to the network. In many cases though, rain streaks can be dense and large in their size, thus to obtain the clean background, we need spatial contextual information. For this, we utilize the dilated convolution. To handle rain accumulation (again, a phenomenon visually similar to mist or fog) and various shapes and directions of overlapping rain streaks, we propose an iterative information feedback (IIF) network that removes rain streaks and clears up the rain accumulation iteratively and progressively. Overall, this multi-task learning and iterative information feedback benefits each other and constitutes a network that is end-to-end trainable. Our extensive evaluation on real images, particularly on heavy rain, shows the effectiveness of our novel models and architecture, outperforming the state-of-the-art methods significantly.

研究动机与目标

• 解决在雨痕密集且雨滴累积效果类似雾或薄雾的强降雨条件下，单幅图像去雨的挑战。
• 通过使用二值分割图显式检测雨痕区域，提升去雨的准确性。
• 通过空洞卷积利用空间上下文信息，增强背景重建效果。
• 通过迭代信息反馈机制，逐步优化预测，实现分阶段去雨。
• 开发一种端到端可训练的架构，联合优化雨痕检测、外观建模与干净背景重建。

提出的方法

• 提出一种改进的图像分解模型，将图像分离为三个分量：二值雨痕图、雨痕外观层与干净背景层。
• 引入一个独立的雨滴累积建模组件，用于表征类似薄雾的视觉雨效，与单个雨痕区分开。
• 采用多任务学习框架，通过共享特征表示，同时预测二值雨图、雨痕外观与干净背景。
• 利用空洞卷积捕捉长距离空间上下文，提升强降雨区域的背景重建质量。
• 设计一种迭代信息反馈（IIF）网络，通过多阶段特征反馈，逐步优化去雨效果。
• 利用二值图的损失作为强监督信号，提升特征学习与检测精度。

实验结果

研究问题

• RQ1 通过二值图显式建模雨痕区域，是否能提升单幅图像去雨网络的性能？
• RQ2 深度学习模型在统一框架下，能否有效处理重叠雨痕与雨滴累积（类似薄雾的效果）？
• RQ3 对去雨预测进行迭代优化，在多大程度上能提升最终干净图像的质量？
• RQ4 采用共享特征的多任务学习（用于检测、外观建模与背景重建）是否能比单任务方法带来更好的泛化能力？
• RQ5 在真实世界强降雨图像上，所提方法与当前最先进方法相比表现如何？

主要发现

• 所提方法在真实世界强降雨图像上优于现有最先进方法，展现出更优的视觉效果与定量指标。
• 将二值雨痕图作为监督任务显著提升了检测精度与特征学习能力，从而增强了整体性能。
• 迭代信息反馈机制实现了雨痕与雨滴累积的逐步去除，促进了更清晰的背景重建。
• 空洞卷积有效捕捉了空间上下文信息，减少了伪影，提升了雨害区域背景的恢复质量。
• 共享特征的多任务架构支持端到端训练，并在多种雨型下实现了更好的泛化能力。
• 该模型对密集且重叠的雨痕以及类似薄雾的雨滴累积均表现出鲁棒性，而这些是先前方法难以处理的挑战。