[Paper Review] Real-World Single Image Super-Resolution: A Brief Review

Honggang Chen, Xiaohai He+4 more|arXiv (Cornell University)|Mar 3, 2021

Advanced Image Processing Techniques136 references22 citations

TL;DR

This paper provides a comprehensive review of real-world single image super-resolution (RSISR), analyzing degradation modeling, image pairs, domain translation, and self-learning methods. It highlights key datasets, evaluation metrics, and challenges such as domain gaps, model efficiency, and lack of reference-free assessment, offering a critical roadmap for advancing practical RSISR applications beyond synthetic data.

ABSTRACT

Single image super-resolution (SISR), which aims to reconstruct a high-resolution (HR) image from a low-resolution (LR) observation, has been an active research topic in the area of image processing in recent decades. Particularly, deep learning-based super-resolution (SR) approaches have drawn much attention and have greatly improved the reconstruction performance on synthetic data. Recent studies show that simulation results on synthetic data usually overestimate the capacity to super-resolve real-world images. In this context, more and more researchers devote themselves to develop SR approaches for realistic images. This article aims to make a comprehensive review on real-world single image super-resolution (RSISR). More specifically, this review covers the critical publically available datasets and assessment metrics for RSISR, and four major categories of RSISR methods, namely the degradation modeling-based RSISR, image pairs-based RSISR, domain translation-based RSISR, and self-learning-based RSISR. Comparisons are also made among representative RSISR methods on benchmark datasets, in terms of both reconstruction quality and computational efficiency. Besides, we discuss challenges and promising research topics on RSISR.

Motivation & Objective

• To address the performance gap between synthetic and real-world image super-resolution by reviewing recent RSISR advancements.
• To identify and analyze the limitations of existing deep learning-based SISR methods when applied to real-world images due to domain shift.
• To summarize publicly available datasets and evaluation metrics critical for training and benchmarking RSISR models.
• To examine four major RSISR method categories: degradation modeling, image pairs, domain translation, and self-learning approaches.
• To highlight open challenges in RSISR, including model efficiency, data scarcity, and the need for no-reference quality assessment.

Proposed method

• Categorizes RSISR methods into four types: degradation modeling-based, image pairs-based, domain translation-based, and self-learning-based approaches.
• Reviews key datasets such as Real-ESRGAN, DIV2K, and others used for training and evaluation of RSISR models.
• Analyzes standard evaluation metrics including PSNR, SSIM, and discusses their limitations in assessing perceptual quality.
• Compares representative RSISR methods on benchmark datasets in terms of reconstruction quality and computational efficiency.
• Proposes a taxonomy of RSISR techniques based on input data type, learning paradigm, and degradation modeling strategy.
• Discusses the importance of realistic degradation modeling and alignment of multi-resolution images for improved performance.

Experimental results

Research questions

• RQ1 How do existing deep learning-based SISR methods perform on real-world images compared to synthetic data?
• RQ2 What are the key differences in degradation patterns between synthetic and real-world images that affect SR performance?
• RQ3 Which RSISR method categories (e.g., degradation modeling, self-supervised learning) show the most promise for real-world deployment?
• RQ4 What are the limitations of current evaluation metrics like PSNR and SSIM in assessing perceptual quality of super-resolved images?
• RQ5 How can RSISR models be made more efficient and adaptable to varying real-world image degradations without paired training data?

Key findings

• Deep learning-based SISR methods achieve high performance on synthetic data but suffer significant performance drops on real-world images due to domain gaps.
• The use of realistic datasets such as Real-ESRGAN and DIV2K has significantly improved RSISR performance, but larger and more diverse datasets are still needed.
• PSNR and SSIM are widely used but fail to correlate well with human perception, especially for real-world images.
• Self-learning-based and domain translation-based RSISR methods show promise in reducing reliance on paired training data.
• Model efficiency and inference speed remain critical challenges, as many state-of-the-art RSISR models are computationally heavy.
• No-reference quality assessment metrics are urgently needed for practical deployment where HR references are unavailable.