[論文レビュー] Multi-Scale Progressive Fusion Network for Single Image Deraining

Kui Jiang, Zhongyuan Wang+6 more|arXiv (Cornell University)|Mar 24, 2020

Image Enhancement Techniques参考文献 41被引用数 52

ひとこと要約

MSPFNは、Gaussian pyramidを介したクロススケール相関、Conv-LSTMベースのcoarse fusion、attention-guided fine fusionを活用して、単一画像のderainingのためのマルチスケール・progressive fusionフレームワークを提案し、最先端の結果を達成するとともに下流の vision タスクを向上させます。

ABSTRACT

Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions, which makes it possible to exploit such complementary information for rain streak representation. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework, termed multi-scale progressive fusion network (MSPFN) for single image rain streak removal. For similar rain streaks at different positions, we employ recurrent calculation to capture the global texture, thus allowing to explore the complementary and redundant information at the spatial dimension to characterize target rain streaks. Besides, we construct multi-scale pyramid structure, and further introduce the attention mechanism to guide the fine fusion of this correlated information from different scales. This multi-scale progressive fusion strategy not only promotes the cooperative representation, but also boosts the end-to-end training. Our proposed method is extensively evaluated on several benchmark datasets and achieves state-of-the-art results. Moreover, we conduct experiments on joint deraining, detection, and segmentation tasks, and inspire a new research direction of vision task-driven image deraining. The source code is available at \url{https://github.com/kuihua/MSPFN}.

研究の動機と目的

• Motivate robust deraining across rain streaks with varied scales and resolutions.
• Propose a unified multi-scale representation that exploits cross-scale and intra-scale rain patterns.
• Develop modules (CFM, FFM, RM) to progressively fuse multi-scale information.
• Demonstrate improved deraining performance and benefits for downstream vision tasks.

提案手法

• Construct Gaussian pyramid rain images to create multi-scale inputs.
• Use a coarse-fusion module (CFM) with Conv-LSTM to capture global texture and cross-scale correlations.
• Apply a fine-fusion module (FFM) with channel attention to selectively fuse multi-scale features.
• Incorporate a reconstruction module (RM) to regresses the residual rain and produce the derained image.
• Train with a loss combining Charbonnier penalty on residual rain and an edge loss on high-frequency details.

実験結果

リサーチクエスチョン

• RQ1 Can multi-scale (cross-scale and intra-scale) rain patterns be effectively exploited to improve single image deraining?
• RQ2 Does progressive fusion across pyramid levels and network stages yield superior deraining quality compared to single-scale or non-progressive multi-scale methods?
• RQ3 Do the proposed MSPFN components (CFM, FFM, RM) contribute positively to both quantitative metrics and visual fidelity?
• RQ4 What is the impact of MSPFN on downstream tasks such as object detection and semantic segmentation when applied to rain-corrupted imagery.

主な発見

• MSPFN achieves state-of-the-art deraining performance on multiple synthetic datasets.
• The multi-scale progressive fusion strategy improves both restoration quality and training efficiency.
• Joint evaluation shows MSPFN enhances downstream detection and segmentation on rain-affected data.
• A lightweight variant of MSPFN maintains competitive deraining while reducing computation for potential mobile/real-time use.