[Paper Review] Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training

Wei Sun, Xiongkuo Min+3 more|arXiv (Cornell University)|May 30, 2021

Image and Video Quality Assessment58 references25 citations

TL;DR

Proposes a BIQA model for in-the-wild images using a staircase feature fusion network and iterative mixed database training to improve generalization across diverse distortions and contents.

ABSTRACT

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA.

Motivation & Objective

• Address the gap that BIQA models trained on synthetic distortions underperform on in-the-wild images.
• Learn quality-aware representations that combine low-, mid-, and high-level features.
• Mitigate limited training data by training across multiple IQA databases and aligning regression heads.
• Demonstrate robustness and generalization through cross-database and six in-the-wild evaluations.

Proposed method

• Introduce a staircase network that hierarchically fuses intermediate CNN layer features to form a quality-aware representation.
• Apply a bottleneck module to align feature map sizes across stages before fusion.
• Merge features progressively across stages to preserve gradient flow and semantic information (Equation 4 and 5 describe fusion process).
• Attach individual image quality regressors for each target database to map features into database-specific quality spaces.
• Train using an iterative mixed database training (IMDT) strategy that alternates optimization across multiple IQA databases while sharing the feature extractor (Algorithm 1).
• Optimize with Euclidean loss between predicted and ground-truth MOS (Equation 6).

Experimental results

Research questions

• RQ1 Can hierarchical fusion of mid-level features improve BIQA performance on authentically distorted images?
• RQ2 Does training on multiple IQA databases with separate regressors yield better generalization than single-database training?
• RQ3 How does IMDT affect feature universality and cross-database robustness?
• RQ4 What are the gains of using a staircase fusion over traditional backbones for BIQA on in-the-wild datasets?

Key findings

• The proposed model achieves state-of-the-art performance on six in-the-wild IQA databases.
• Cross-database evaluations show robust generalization of the learned quality-aware features.
• Hierarchical feature fusion (staircase network) better captures low-, mid-, and high-level information for quality assessment.
• IMDT increases training sample diversity and improves feature generalization by training across multiple databases with database-specific regressors.
• On six in-the-wild datasets, the proposed method outperforms competing BIQA models by a large margin.
• Code for StairIQA is released at the project repository.