[Paper Review] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Xintao Wang, Liangbin Xie+2 more|ArXiv.org|Jul 22, 2021

Advanced Image Processing TechniquesComputer Science52 references83 citations

TL;DR

Real-ESRGAN extends ESRGAN by using a high-order synthetic degradation model and a U-Net discriminator with spectral normalization to train on pure synthetic data, achieving superior real-world blind super-resolution results.

ABSTRACT

Though many attempts have been made in blind super-resolution to restore low-resolution images with unknown and complex degradations, they are still far from addressing general real-world degraded images. In this work, we extend the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data. Specifically, a high-order degradation modeling process is introduced to better simulate complex real-world degradations. We also consider the common ringing and overshoot artifacts in the synthesis process. In addition, we employ a U-Net discriminator with spectral normalization to increase discriminator capability and stabilize the training dynamics. Extensive comparisons have shown its superior visual performance than prior works on various real datasets. We also provide efficient implementations to synthesize training pairs on the fly.

Motivation & Objective

• Motivate real-world blind SR where degradations are unknown and complex.
• Develop a practical degradation model that better simulates real-world processes.
• Enhance training stability and perceptual quality through architectural and training enhancements.
• Demonstrate that synthetic-data training can outperform prior real-data and synthetic-data methods for real-world images.

Proposed method

• Introduce a high-order degradation model that composes multiple classical degradations to simulate real-world pipelines.
• Incorporate a sinc filter to model ringing and overshoot artifacts during degradation synthesis.
• Adopt a U-Net discriminator with spectral normalization to provide stronger, per-pixel feedback and stabilize GAN training.
• Use a two-stage training scheme: first train a PSNR-oriented model (Real-ESRNet), then fine-tune with L1, perceptual, and GAN losses.
• Extend ESRGAN architecture to handle scale factors of 2 and 1 via pixel-unshuffle before processing in the RRDB-based generator.

Experimental results

Research questions

• RQ1 Can a high-order synthetic degradation model bridge the gap between synthetic and real-world degradations for blind SR?
• RQ2 Does incorporating sinc-based ringing artifacts and a U-Net discriminator improve real-world restoration quality and training stability?
• RQ3 Is synthetic data alone sufficient to surpass prior real-world SR methods on diverse real datasets?
• RQ4 Does sharpening ground-truths during training further improve perceptual quality without introducing artifacts?
• RQ5 How does Real-ESRGAN compare to ESRGAN and other state-of-the-art blind SR methods on real-world imagery?

Key findings

• Real-ESRGAN trained with pure synthetic data yields better visual performance than prior methods on real datasets.
• A second-order degradation model plus sinc-based artifacts helps simulate real-world degradations and reduces ringing/overshoot in outputs.
• A U-Net discriminator with spectral normalization improves local texture reconstruction and stabilizes GAN training.
• Real-ESRGAN with sharpened ground-truths further improves perceived sharpness without introducing obvious artifacts.
• The two-stage training (Real-ESRNet initialization followed by Real-ESRGAN training) accelerates convergence and improves results.