[Paper Review] Restoring Images with Unknown Degradation Factors by Recurrent Use of a Multi-branch Network

Xing Liu, Masanori Suganuma+2 more|arXiv (Cornell University)|Jul 10, 2019

Advanced Image Processing Techniques72 references4 citations

TL;DR

This paper proposes a recurrent multi-branch network that restores images with unknown degradation factors by iteratively applying different restoration branches to the same shared backbone. The method achieves performance comparable or superior to dedicated networks on single degradation types and outperforms prior state-of-the-art on mixed degradation tasks, demonstrating strong generalization with a two-step training scheme combining multi-task learning and fine-tuning.

ABSTRACT

The employment of convolutional neural networks has achieved unprecedented performance in the task of image restoration for a variety of degradation factors. However, high-performance networks have been specifically designed for a single degradation factor. In this paper, we tackle a harder problem, restoring a clean image from its degraded version with an unknown degradation factor, subject to the condition that it is one of the known factors. Toward this end, we design a network having multiple pairs of input and output branches and use it in a recurrent fashion such that a different branch pair is used at each of the recurrent paths. We reinforce the shared part of the network with improved components so that it can handle different degradation factors. We also propose a two-step training method for the network, which consists of multi-task learning and finetuning. The experimental results show that the proposed network yields at least comparable or sometimes even better performance on four degradation factors as compared with the best dedicated network for each of the four. We also test it on a further harder task where the input image contains multiple degradation factors that are mixed with unknown mixture ratios, showing that it achieves better performance than the previous state-of-the-art method designed for the task.

Motivation & Objective

• To address image restoration when the degradation factor is unknown but known to be one of a predefined set.
• To overcome the limitations of dedicated networks that require prior knowledge of degradation type.
• To design a single architecture that generalizes across multiple degradation types without significant performance loss.
• To enable effective restoration of images with mixed, unknown degradation ratios, a more realistic and challenging scenario.
• To develop a training strategy that maintains performance across diverse degradation types while minimizing parameter overhead.

Proposed method

• The network employs multiple input-output branch pairs, each specialized for a specific degradation type, connected through a shared backbone.
• A recurrent mechanism reuses the same network by feeding the output of one branch pair back into a different input branch in subsequent iterations.
• The shared backbone is enhanced with a new residual block design, DuRB-M, to improve representational capacity across diverse degradation types.
• A two-step training process is used: first multi-task learning across all degradation types, followed by fine-tuning on mixed degradation data to improve robustness.
• The network predicts residual images relative to the input, enabling end-to-end learning with consistent residual learning formulation per degradation type.
• The architecture is trained to minimize degradation-specific losses while preserving clean images and avoiding degradation of non-target factors.

Experimental results

Research questions

• RQ1 Can a single deep network effectively restore images when the degradation type is unknown but limited to a known set of factors?
• RQ2 How can a shared network backbone be designed to generalize across multiple distinct degradation types without performance degradation?
• RQ3 Does a recurrent application of multi-branch network branches improve restoration performance compared to single-pass or cascaded approaches?
• RQ4 Can a two-step training strategy—multi-task learning followed by fine-tuning—enhance generalization on mixed degradation scenarios?
• RQ5 How does the proposed method compare to dedicated networks and prior state-of-the-art methods on both single and combined degradation tasks?

Key findings

• The proposed R-MBN model achieves performance on par with or better than dedicated networks for each of the four individual degradation types (blur, noise, JPEG, haze).
• The R-MBN* variant, fine-tuned on mixed degradation data, outperforms the prior state-of-the-art method of Suganuma et al. on combined degradation tasks with large margins.
• Even without fine-tuning on mixed data, R-MBN maintains strong performance on mixed degradations, with only a moderate drop compared to its performance on single factors.
• The method shows robustness to unknown mixture ratios of degradation factors, indicating strong generalization capability.
• The two-step training strategy effectively balances multi-task learning and fine-tuning, enabling the network to handle diverse degradation patterns while preserving image quality.