[Paper Review] Re-IQA: Unsupervised Learning for Image Quality Assessment in the Wild

Avinab Saha, Sandeep Mishra+1 more|arXiv (Cornell University)|Apr 2, 2023

Image and Video Quality Assessment4 citations

TL;DR

This paper proposes Re-IQA, a novel unsupervised learning framework that jointly learns complementary high-level content and low-level image quality representations using a Mixture of Experts setup. By training two separate encoders via contrastive learning on overlapping image patches and diverse distortions, Re-IQA achieves state-of-the-art performance on multiple no-reference image quality assessment benchmarks, demonstrating that unsupervised quality-aware features significantly improve linear regressor performance.

ABSTRACT

Automatic Perceptual Image Quality Assessment is a challenging problem that impacts billions of internet, and social media users daily. To advance research in this field, we propose a Mixture of Experts approach to train two separate encoders to learn high-level content and low-level image quality features in an unsupervised setting. The unique novelty of our approach is its ability to generate low-level representations of image quality that are complementary to high-level features representing image content. We refer to the framework used to train the two encoders as Re-IQA. For Image Quality Assessment in the Wild, we deploy the complementary low and high-level image representations obtained from the Re-IQA framework to train a linear regression model, which is used to map the image representations to the ground truth quality scores, refer Figure 1. Our method achieves state-of-the-art performance on multiple large-scale image quality assessment databases containing both real and synthetic distortions, demonstrating how deep neural networks can be trained in an unsupervised setting to produce perceptually relevant representations. We conclude from our experiments that the low and high-level features obtained are indeed complementary and positively impact the performance of the linear regressor. A public release of all the codes associated with this work will be made available on GitHub.

Motivation & Objective

• To develop an unsupervised framework that learns perceptually relevant, complementary low-level image quality and high-level content representations for no-reference image quality assessment.
• To address the challenge of accurate image quality prediction in real-world scenarios where reference images are unavailable and distortions are diverse and content-dependent.
• To demonstrate that unsupervised pre-training on authentic, distorted images can yield high-quality features competitive with supervised baselines.
• To validate the effectiveness of the Mixture of Experts approach in disentangling content and quality representations for improved IQA performance.

Proposed method

• A Mixture of Experts framework trains two separate encoders: one for high-level content features using MoCo-v2 contrastive learning, and another for low-level quality-aware features.
• The quality-aware encoder uses a contrastive learning objective based on overlapping image patches to learn perceptual quality-aware features (PQAF), assuming local similarity in quality perception.
• Image patches from the same image are used as positive pairs, while patches from different images are treated as negative pairs, enforcing discriminative quality representation learning.
• The framework assumes three key hypotheses: (1) quality perception varies locally within images based on content, (2) images from different sources have different quality, and (3) different distortions of the same image yield different perceptual quality scores.
• After pre-training, the frozen encoders extract features that are fed into a linear regressor to predict ground-truth quality scores from databases like KonIQ, AVA, and LIVE-IQA.
• The method is evaluated in both no-reference (Re-IQA-NR) and full-reference (Re-IQA-FR) settings, with ablation studies on patch overlap and model components.

Experimental results

Research questions

• RQ1 Can unsupervised contrastive learning effectively extract low-level, quality-aware image representations that are complementary to high-level content features?
• RQ2 Does the joint use of content and quality representations improve no-reference image quality assessment performance compared to using either alone?
• RQ3 How does unsupervised pre-training on real-world distorted images compare to supervised ImageNet pre-training for IQA representation learning?
• RQ4 To what extent do the proposed hypotheses about perceptual quality variation within and across images support effective representation learning?

Key findings

• Re-IQA achieves state-of-the-art performance on multiple large-scale IQA databases, including KonIQ, AVA, and LIVE-IQA, with Spearman’s rank correlation (SRCC) scores exceeding 0.85 on KonIQ and AVA.
• The Re-IQA-FR variant, which uses reference images, achieves slightly better SRCC and PLCC than Re-IQA-NR, confirming the benefit of additional reference information.
• The t-SNE visualization of quality-aware features shows clear clustering of high- and low-quality images, with minimal overlap, indicating effective separation of perceptual quality levels.
• Ablation studies confirm that patch overlap degree significantly affects performance, with optimal results achieved at moderate overlap, validating the local quality perception hypothesis.
• The unsupervised ResNet-50 encoder pre-trained via MoCo-v2 outperforms the supervised ImageNet-pretrained ResNet-50 in content representation quality for IQA, highlighting the benefit of self-supervised pre-training on real distortions.