[Paper Review] NTIRE 2022 Challenge on Perceptual Image Quality Assessment

Jinjin Gu, Haoming Cai+3 more|arXiv (Cornell University)|Jun 23, 2022

Image and Video Quality Assessment4 citations

TL;DR

This paper presents the NTIRE 2022 challenge on perceptual image quality assessment, introducing a novel benchmark using the PIPAL dataset to evaluate full-reference and no-reference IQA methods on GAN-based and perceptual-oriented distortions. The winning models achieved state-of-the-art performance, demonstrating superior correlation with human judgments across diverse distortion types using advanced architectures and loss functions.

ABSTRACT

This paper reports on the NTIRE 2022 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2022. This challenge is held to address the emerging challenge of IQA by perceptual image processing algorithms. The output images of these algorithms have completely different characteristics from traditional distortions and are included in the PIPAL dataset used in this challenge. This challenge is divided into two tracks, a full-reference IQA track similar to the previous NTIRE IQA challenge and a new track that focuses on the no-reference IQA methods. The challenge has 192 and 179 registered participants for two tracks. In the final testing stage, 7 and 8 participating teams submitted their models and fact sheets. Almost all of them have achieved better results than existing IQA methods, and the winning method can demonstrate state-of-the-art performance.

Motivation & Objective

• Address the growing challenge of evaluating perceptual image quality in modern image processing pipelines, especially those using GAN-based and perceptual-oriented algorithms.
• Benchmark existing and novel IQA methods on a large-scale, diverse dataset containing real-world distortions from advanced image restoration algorithms.
• Advance no-reference IQA by introducing a dedicated track to stimulate innovation in blind quality assessment for complex, non-traditional distortions.
• Provide a standardized evaluation platform to compare and validate new IQA models against human perception metrics.
• Drive progress in perceptual image quality assessment by fostering community-wide competition and collaboration through a structured challenge framework.

Proposed method

• Utilized the PIPAL dataset, which contains 250 reference images, 29,000 distorted images, and 1.88 million human judgments, to train and evaluate IQA models.
• Designed two distinct tracks: full-reference IQA (comparing distorted images to a reference) and no-reference IQA (evaluating distorted images without a reference).
• Employed deep learning architectures such as Transformers, CNNs, and hybrid networks to model perceptual quality, with attention mechanisms enhancing feature representation.
• Applied multi-loss training strategies including Pearson correlation loss, PLCC loss, and triplet ranking loss to align model predictions with human judgments.
• Used AdamW optimizer with a base learning rate of 0.0001 and a cosine decay schedule over 50 epochs, with a batch size of 10.
• Incorporated ensemble methods and data augmentation techniques to improve generalization and robustness across diverse distortion types.

Experimental results

Research questions

• RQ1 Can deep learning-based IQA models effectively predict human perception of image quality in the presence of modern, GAN-based distortions?
• RQ2 How do no-reference IQA models perform on perceptual distortions compared to traditional full-reference methods?
• RQ3 What architectural and loss function designs lead to the highest correlation with human subjective judgments on complex, real-world distortions?
• RQ4 To what extent do ensemble methods and multi-loss training improve IQA model generalization and performance on unseen distortion types?
• RQ5 Can the proposed challenge framework serve as a reliable benchmark for future development of perceptual IQA methods?

Key findings

• The winning full-reference method (THU1919Group) achieved a PIPAL-NTIRE22-Test score of 1.6511, with SRCC of 0.8227 and PLCC of 0.8284, significantly outperforming baseline methods.
• The top no-reference method (THU_IIGROUP) achieved a score of 1.4436, with SRCC 0.7040 and PLCC 0.7396, demonstrating strong performance without reference images.
• All top-performing models surpassed existing baselines such as IQT (NTIRE-21 winner), LPIPS, DISTS, SSIM, and PSNR in both tracks.
• The use of attention mechanisms and multi-loss training significantly improved correlation with human judgments, particularly in capturing subtle perceptual differences.
• The challenge attracted 192 and 179 registered participants for the full-reference and no-reference tracks, respectively, with 7 and 8 teams submitting final models.
• The results confirm that modern deep learning architectures with careful loss design can achieve state-of-the-art performance on perceptual IQA, especially on novel distortion types.