[Paper Review] Unsupervised Image Super-Resolution using Cycle-in-Cycle Generative Adversarial Networks

Yuan Yuan, Siyuan Liu+4 more|arXiv (Cornell University)|Sep 3, 2018

Advanced Image Processing TechniquesComputer Science27 references40 citations

TL;DR

Introduces CinCGAN, a Cycle-in-Cycle GAN framework for unsupervised single image super-resolution when LR-HR pairs and degradation kernels are unavailable, achieving competitive results with supervised methods.

ABSTRACT

We consider the single image super-resolution problem in a more general case that the low-/high-resolution pairs and the down-sampling process are unavailable. Different from traditional super-resolution formulation, the low-resolution input is further degraded by noises and blurring. This complicated setting makes supervised learning and accurate kernel estimation impossible. To solve this problem, we resort to unsupervised learning without paired data, inspired by the recent successful image-to-image translation applications. With generative adversarial networks (GAN) as the basic component, we propose a Cycle-in-Cycle network structure to tackle the problem within three steps. First, the noisy and blurry input is mapped to a noise-free low-resolution space. Then the intermediate image is up-sampled with a pre-trained deep model. Finally, we fine-tune the two modules in an end-to-end manner to get the high-resolution output. Experiments on NTIRE2018 datasets demonstrate that the proposed unsupervised method achieves comparable results as the state-of-the-art supervised models.

Motivation & Objective

• Address the lack of paired LR-HR data and unknown degradation in SR tasks.
• Propose an unsupervised training strategy inspired by image-to-image translation.
• Develop a Cycle-in-Cycle architecture to denoise/deblur at LR before upsampling.
• Fine-tune the combined LR-to-HR pipeline end-to-end for high-quality SR.

Proposed method

• Use two coupled CycleGANs to map LR to clean LR and then to HR, without paired data.
• First CycleGAN (LR to clean LR) includes denoising/deblurring with cycle, identity, and TV losses.
• Stack a pre-trained SR model (EDSR) after the LR-to-clean-LR module for upsampling.
• Jointly fine-tune LR-to-clean-LR and LR-to-HR paths with adversarial and cycle-consistency losses.
• Adopt a PatchGAN discriminator for the HR stage and introduce an SR identity loss to stabilize SR quality.
• Train with unsupervised losses and then fine-tune end-to-end on NTIRE2018 track 2 data.

Experimental results

Research questions

• RQ1 Can unsupervised learning with Cycle-in-Cycle GANs achieve competitive SR performance without paired data?
• RQ2 Does decomposing denoising/deblurring at LR before upsampling improve stability and quality over single-stage unsupervised SR?
• RQ3 How does CinCGAN compare to supervised SR models and denoising-then-SR pipelines on real degradation datasets?

Key findings

• CinCGAN achieves competitive PSNR/SSIM with state-of-the-art supervised models on NTIRE2018 track 2 data.
• On average, CinCGAN attains PSNR/SSIM of 24.33/0.69, comparable to SRGAN+ (24.33/0.68) but lower than EDSR+ in PSNR (25.77/0.71).
• Directly applying supervised SR models without paired data yields poorer results under unknown degradations; CinCGAN mitigates this via LR-to-clean-LR denoising in an unsupervised manner.
• Ablation studies show that removing components degradations or cycle constraints leads to noisier, less stable SR results, validating the CinCGAN structure.
• BM3D+EDSR denoises first but remains less effective than CinCGAN, indicating joint unsupervised denoising and upsampling is beneficial.