[Paper Review] Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration

Xing Liu, Masanori Suganuma+2 more|arXiv (Cornell University)|Mar 21, 2019

Advanced Image Processing Techniques58 references17 citations

TL;DR

This paper proposes Dual Residual Networks (DuRN) with a novel 'dual residual connection' mechanism that enables interactions between paired operations—such as up/down-sampling or large/small kernel convolutions—across different blocks in a residual network. By using a modular Dual Residual Block (DuRB) with two configurable operation containers, the architecture exploits a broader set of implicit paths, achieving state-of-the-art performance across five image restoration tasks on nine datasets without adversarial training or recurrence.

ABSTRACT

In this paper, we study design of deep neural networks for tasks of image restoration. We propose a novel style of residual connections dubbed "dual residual connection", which exploits the potential of paired operations, e.g., up- and down-sampling or convolution with large- and small-size kernels. We design a modular block implementing this connection style; it is equipped with two containers to which arbitrary paired operations are inserted. Adopting the "unraveled" view of the residual networks proposed by Veit et al., we point out that a stack of the proposed modular blocks allows the first operation in a block interact with the second operation in any subsequent blocks. Specifying the two operations in each of the stacked blocks, we build a complete network for each individual task of image restoration. We experimentally evaluate the proposed approach on five image restoration tasks using nine datasets. The results show that the proposed networks with properly chosen paired operations outperform previous methods on almost all of the tasks and datasets.

Motivation & Objective

• To improve deep neural network architectures for image restoration by leveraging the potential of paired operations such as up/down-sampling or large/small kernel convolutions.
• To design a new residual connection style that enables cross-block interaction between paired operations, increasing the diversity of implicit paths in residual networks.
• To develop a modular building block (DuRB) that supports arbitrary paired operations and enables flexible network construction across multiple image restoration tasks.
• To empirically validate the effectiveness of the proposed architecture on diverse image restoration benchmarks, demonstrating consistent performance gains.

Proposed method

• Proposes a new residual connection style called 'dual residual connection', allowing any first operation in an early block to pair with any second operation in a later block.
• Designs a Dual Residual Block (DuRB) with two containers for inserting paired operations, enabling flexible and reusable network design.
• Adopts the 'unraveled' view of residual networks (Veit et al.) to analyze implicit paths, showing that dual residual connections increase the number of effective operation pairs across blocks.
• Constructs task-specific networks (DuRN) by stacking DuRBs and selecting appropriate paired operations—e.g., dilated convolutions for rain removal, symmetric convolutions for denoising.
• Employs a weighted sum of SSIM and L1 loss for training, avoiding adversarial or recurrent components to maintain training stability.
• Uses encoder-decoder structures with DuRB-S (for attention-based localization) and DuRB-P (for receptive field expansion) in hybrid designs for rain removal.

Experimental results

Research questions

• RQ1 Can a novel residual connection style that enables cross-block pairing of operations improve image restoration performance?
• RQ2 How does the increased number of implicit paths from dual residual connections affect feature learning and task performance?
• RQ3 Can a modular block (DuRB) with two configurable operation containers generalize across diverse image restoration tasks?
• RQ4 Does the proposed architecture outperform existing SOTA methods without using adversarial training or recurrent modules?
• RQ5 What specific paired operation combinations yield optimal performance for different restoration tasks?

Key findings

• The proposed DuRN-S-P network outperforms the state-of-the-art method of Qian et al. [32] on the RainDrop Dataset, achieving 31.24 dB PSNR and 0.9259 SSIM on TestSetA.
• On the DDN-Data and DID-MDN-Data benchmarks for rain removal, the DuRN-S model achieves 31.30 dB and 33.21 dB PSNR, respectively, surpassing previous SOTA methods including RESCAN and NLEDN.
• The DuRN-S-P network achieves 25.32 dB PSNR and 0.8173 SSIM on TestSetB of the RainDrop Dataset, demonstrating strong generalization on more challenging test samples.
• The method achieves consistent performance gains across five image restoration tasks—Gaussian noise removal, motion blur removal, rain streak removal, super-resolution, and JPEG artifact reduction—on nine datasets.
• The proposed architecture achieves these results without using adversarial training or recurrent networks, highlighting the effectiveness of the dual residual connection design alone.
• The ablation study confirms that the dual residual connection mechanism enables richer feature interactions than standard residual or paired-block designs, leading to improved performance.