[Paper Review] Multi-Scale Progressive Fusion Network for Single Image Deraining

Kui Jiang, Zhongyuan Wang+6 more|arXiv (Cornell University)|Mar 24, 2020

Image Enhancement Techniques41 references52 citations

TL;DR

MSPFN introduces a multi-scale, progressive fusion framework for single image deraining that leverages cross-scale correlations via a Gaussian pyramid, Conv-LSTM based coarse fusion, and attention-guided fine fusion to achieve state-of-the-art results and boost downstream vision tasks.

ABSTRACT

Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions, which makes it possible to exploit such complementary information for rain streak representation. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework, termed multi-scale progressive fusion network (MSPFN) for single image rain streak removal. For similar rain streaks at different positions, we employ recurrent calculation to capture the global texture, thus allowing to explore the complementary and redundant information at the spatial dimension to characterize target rain streaks. Besides, we construct multi-scale pyramid structure, and further introduce the attention mechanism to guide the fine fusion of this correlated information from different scales. This multi-scale progressive fusion strategy not only promotes the cooperative representation, but also boosts the end-to-end training. Our proposed method is extensively evaluated on several benchmark datasets and achieves state-of-the-art results. Moreover, we conduct experiments on joint deraining, detection, and segmentation tasks, and inspire a new research direction of vision task-driven image deraining. The source code is available at \url{https://github.com/kuihua/MSPFN}.

Motivation & Objective

• Motivate robust deraining across rain streaks with varied scales and resolutions.
• Propose a unified multi-scale representation that exploits cross-scale and intra-scale rain patterns.
• Develop modules (CFM, FFM, RM) to progressively fuse multi-scale information.
• Demonstrate improved deraining performance and benefits for downstream vision tasks.

Proposed method

• Construct Gaussian pyramid rain images to create multi-scale inputs.
• Use a coarse-fusion module (CFM) with Conv-LSTM to capture global texture and cross-scale correlations.
• Apply a fine-fusion module (FFM) with channel attention to selectively fuse multi-scale features.
• Incorporate a reconstruction module (RM) to regresses the residual rain and produce the derained image.
• Train with a loss combining Charbonnier penalty on residual rain and an edge loss on high-frequency details.

Experimental results

Research questions

• RQ1 Can multi-scale (cross-scale and intra-scale) rain patterns be effectively exploited to improve single image deraining?
• RQ2 Does progressive fusion across pyramid levels and network stages yield superior deraining quality compared to single-scale or non-progressive multi-scale methods?
• RQ3 Do the proposed MSPFN components (CFM, FFM, RM) contribute positively to both quantitative metrics and visual fidelity?
• RQ4 What is the impact of MSPFN on downstream tasks such as object detection and semantic segmentation when applied to rain-corrupted imagery.

Key findings

• MSPFN achieves state-of-the-art deraining performance on multiple synthetic datasets.
• The multi-scale progressive fusion strategy improves both restoration quality and training efficiency.
• Joint evaluation shows MSPFN enhances downstream detection and segmentation on rain-affected data.
• A lightweight variant of MSPFN maintains competitive deraining while reducing computation for potential mobile/real-time use.