[Paper Review] ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks

Xintao Wang, Ke Yu+7 more|arXiv (Cornell University)|Sep 1, 2018

Advanced Image Processing Techniques40 references1,093 citations

TL;DR

ESRGAN improves SRGAN by introducing Residual-in-Residual Dense Blocks without batch normalization, a relativistic discriminator, and perceptual loss computed on features before activation, achieving superior perceptual quality and winning PIRM-SR Challenge region 3.

ABSTRACT

The Super-Resolution Generative Adversarial Network (SRGAN) is a seminal work that is capable of generating realistic textures during single image super-resolution. However, the hallucinated details are often accompanied with unpleasant artifacts. To further enhance the visual quality, we thoroughly study three key components of SRGAN - network architecture, adversarial loss and perceptual loss, and improve each of them to derive an Enhanced SRGAN (ESRGAN). In particular, we introduce the Residual-in-Residual Dense Block (RRDB) without batch normalization as the basic network building unit. Moreover, we borrow the idea from relativistic GAN to let the discriminator predict relative realness instead of the absolute value. Finally, we improve the perceptual loss by using the features before activation, which could provide stronger supervision for brightness consistency and texture recovery. Benefiting from these improvements, the proposed ESRGAN achieves consistently better visual quality with more realistic and natural textures than SRGAN and won the first place in the PIRM2018-SR Challenge. The code is available at https://github.com/xinntao/ESRGAN .

Motivation & Objective

• Motivate improving perceptual quality in single image super-resolution (SISR) beyond PSNR-oriented methods.
• Develop a deeper generator architecture that is easier to train and yields richer textures.
• Incorporate a more effective discriminator and perceptual loss to guide texture recovery.

Proposed method

• Replace batch normalization with Residual-in-Residual Dense Blocks (RRDB) to form a deeper generator.
• Remove BN layers, apply residual scaling, and use smaller initialization for stable GAN training.
• Adopt Relativistic average Discriminator (RaD) to train the generator with relativistic judgments.
• Compute perceptual loss using features before activation to provide stronger, brightness-preserving supervision.
• Optionally include a MINC-based perceptual loss variant focused on textures.
• Introduce network interpolation to balance perceptual quality and fidelity without retraining.

Experimental results

Research questions

• RQ1 How does removing batch normalization and using RRDBs affect training stability and visual quality in SRGAN-based models?
• RQ2 Does a relativistic discriminator improve texture realism and edge sharpness in super-resolved outputs compared to a standard GAN discriminator?
• RQ3 Does using pre-activation features for perceptual loss yield brighter, sharper results than post-activation features, and how does this affect texture recovery?
• RQ4 Can network interpolation effectively balance perceptual quality with PSNR/structure fidelity without additional training?
• RQ5 What gains do additional datasets and training strategies provide for realism and texture detail in SR under various evaluation regimes (e.g., PIRM-SR)?

Key findings

• ESRGAN yields consistently better perceptual quality and textures than SRGAN and other PSNR-oriented methods.
• Removing BN layers and adopting RRDBs enables training of a deeper generator with improved texture recovery.
• Relativistic average discriminator improves edge sharpness and detail in generated textures.
• Perceptual loss computed on pre-activation features provides sharper edges and more accurate brightness than post-activation features.
• Network interpolation offers smooth control to balance perceptual quality and fidelity without retraining, outperforming simple image interpolation.
• The model variant achieved first place in the PIRM-SR Challenge region 3 with the best perceptual index.