[Paper Review] Feedback Network for Image Super-Resolution

Zhen Li, Jinglei Yang+4 more|arXiv (Cornell University)|Mar 23, 2019

Advanced Image Processing Techniques47 references47 citations

TL;DR

The paper introduces SRFBN, a recurrent feedback network for image super-resolution that refines low-level features with high-level information via a dedicated feedback block, plus curriculum learning for complex degradations.

ABSTRACT

Recent advances in image super-resolution (SR) explored the power of deep learning to achieve a better reconstruction performance. However, the feedback mechanism, which commonly exists in human visual system, has not been fully exploited in existing deep learning based image SR methods. In this paper, we propose an image super-resolution feedback network (SRFBN) to refine low-level representations with high-level information. Specifically, we use hidden states in an RNN with constraints to achieve such feedback manner. A feedback block is designed to handle the feedback connections and to generate powerful high-level representations. The proposed SRFBN comes with a strong early reconstruction ability and can create the final high-resolution image step by step. In addition, we introduce a curriculum learning strategy to make the network well suitable for more complicated tasks, where the low-resolution images are corrupted by multiple types of degradation. Extensive experimental results demonstrate the superiority of the proposed SRFBN in comparison with the state-of-the-art methods. Code is avaliable at https://github.com/Paper99/SRFBN_CVPR19.

Motivation & Objective

• Motivate the use of a feedback mechanism to refine low-level representations using high-level information in SR.
• Design a feedback block that efficiently handles top-down information flow and enriches high-level representations.
• Enable strong early reconstruction and gradual refinement across iterations with a curriculum learning strategy for complex degradations.

Proposed method

• Propose SRFBN, an iterative network unfolded for T iterations where each iteration contains an LR feature block, a feedback block, and a reconstruction block.
• Introduce a feedback block (FB) with multiple projection groups and dense skip connections to propagate high-level information back to low-level features.
• Tie the loss across all iterations to enforce useful high-level information in hidden states.
• Use a Curriculum Learning strategy to present targets from easy to hard across iterations for complex degradation models BD and DN.

Experimental results

Research questions

• RQ1 Can a feedback mechanism improve low-level feature refinement in SR by routing high-level information back to earlier stages?
• RQ2 Does a dedicated feedback block with dense projections enhance reconstruction quality with fewer parameters?
• RQ3 Does curriculum learning improve SR performance under multiple degradation models compared to standard training?
• RQ4 How do early iteration outputs compare to later ones, and what is the benefit of multi-iteration supervision in SR?

Key findings

• SRFBN with feedback outperforms its feedforward counterpart and improves as the number of iterations T increases.
• Larger projection groups G in the feedback block yield better SR accuracy, highlighting richer high-level representations.
• SRFBN-S achieves strong SR results among lightweight models (under 1M parameters).
• Self-ensemble version SRFBN+ yields further improvements, approaching or surpassing some larger models while maintaining fewer parameters.
• Curriculum learning enhances SR performance under BD and DN degradations, and fine-tuning from BI helps further boost results.