[Paper Review] Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution

Yong Guo, Jian Chen+6 more|arXiv (Cornell University)|Mar 16, 2020

Advanced Image Processing TechniquesComputer Science51 references32 citations

TL;DR

The paper proposes Dual Regression Networks (DRN) for single image super-resolution, introducing a dual regression constraint that reconstructs LR from HR to form a closed loop, enabling better SR and adaptation to unpaired real-world data.

ABSTRACT

Deep neural networks have exhibited promising performance in image super-resolution (SR) by learning a nonlinear mapping function from low-resolution (LR) images to high-resolution (HR) images. However, there are two underlying limitations to existing SR methods. First, learning the mapping function from LR to HR images is typically an ill-posed problem, because there exist infinite HR images that can be downsampled to the same LR image. As a result, the space of the possible functions can be extremely large, which makes it hard to find a good solution. Second, the paired LR-HR data may be unavailable in real-world applications and the underlying degradation method is often unknown. For such a more general case, existing SR models often incur the adaptation problem and yield poor performance. To address the above issues, we propose a dual regression scheme by introducing an additional constraint on LR data to reduce the space of the possible functions. Specifically, besides the mapping from LR to HR images, we learn an additional dual regression mapping estimates the down-sampling kernel and reconstruct LR images, which forms a closed-loop to provide additional supervision. More critically, since the dual regression process does not depend on HR images, we can directly learn from LR images. In this sense, we can easily adapt SR models to real-world data, e.g., raw video frames from YouTube. Extensive experiments with paired training data and unpaired real-world data demonstrate our superiority over existing methods.

Motivation & Objective

• Motivate SR as an ill-posed problem with infinite possible LR→HR mappings.
• Reduce the function space of SR mappings via a dual (LR reconstruction) constraint.
• Enable learning from unpaired real-world LR data without HR counterparts.
• Provide a training framework that combines paired synthetic data with unpaired real-world data.

Proposed method

• Introduce primal (LR→HR) and dual (HR→LR) regression tasks forming a closed loop.
• Use a U-Net–based primal network with RCAB blocks for SR reconstruction.
• Implement a lightweight dual network to learn downsampling as HR→LR reconstruction.
• Optimize a joint loss: primal SR loss plus a weighted dual reconstruction loss (lambda).
• Extend to unpaired data via an adaptation algorithm that blends paired synthetic data with real-world LR data.
• Provide theoretical generalization analysis based on Rademacher complexity.

Experimental results

Research questions

• RQ1 Can a dual regression constraint reduce the space of LR→HR mappings to improve SR performance?
• RQ2 Can learning a LR reconstruction mapping enable effective adaptation to unpaired real-world LR data?
• RQ3 How does DRN perform on paired SR benchmarks versus unpaired real-world data adaptation?
• RQ4 What is the impact of the dual loss weight lambda and data ratio rho on performance?
• RQ5 Do DRN models offer theoretical generalization benefits over traditional supervised SR?

Key findings

• DRN with dual regression yields sharper, more accurate SR results than several state-of-the-art baselines on paired data, especially for 8× SR.
• The dual regression constraint reduces the function space and improves SR reconstruction quality.
• DRN can adapt to real-world unpaired LR data, achieving superior results over CycleGAN-based methods and other baselines in adaptation scenarios.
• The adaptation algorithm effectively leverages a mix of unpaired real-world data and paired synthetic data, with a recommended unpaired data ratio (rho) around 30%.
• A theoretical generalization bound suggests the dual regression scheme has a smaller bound than conventional supervised SR, supporting better generalization under sufficient data and model capacity.