[论文解读] Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training

Wei Sun, Xiongkuo Min+3 more|arXiv (Cornell University)|May 30, 2021

Image and Video Quality Assessment参考文献 58被引用 25

一句话总结

提出一项用于野外图像的 BIQA 模型，通过阶梯特征融合网络和迭代混合数据库训练来提高对多样失真和内容的泛化。

ABSTRACT

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA.

研究动机与目标

• 解决在合成失真上训练的 BIQA 模型在野外图像上的性能不足的问题。
• 学习结合低-中-高层特征的质量感知表示。
• 通过跨多个 IQA 数据库进行训练并对齐回归头，缓解训练数据有限的问题。
• 通过跨数据库和六个野外评测展示鲁棒性和泛化能力。

提出的方法

• 提出一个阶梯网络，在层级地融合中间 CNN 层的特征以形成质量感知表示。
• 应用瓶颈模块在融合前对齐不同阶段的特征图大小。
• 在各阶段逐步合并特征以保留梯度流和语义信息（方程式 4 和 5 描述融合过程）。
• 为每个目标数据库附加独立的图像质量回归器，将特征映射到数据库特定的质量空间。
• 使用迭代混合数据库训练（IMDT）策略进行训练，在共享特征提取器的同时在多个 IQA 数据库之间交替优化（算法 1）。
• 使用预测值与真实 MOS 之间的欧氏损失进行优化（方程式 6）。

实验结果

研究问题

• RQ1 分层融合中等层特征是否能提高对真实失真图像的 BIQA 性能？
• RQ2 在多个 IQA 数据库上使用单独回归器进行训练是否比单一数据库训练具有更好的泛化能力？
• RQ3 IMDT 如何影响特征的通用性和跨数据库的鲁棒性？
• RQ4 在野外数据集上使用阶梯融合相对于传统骨干网络的 BIQA 能够带来哪些提升？

主要发现

• 所提出的模型在六个野外 IQA 数据库上达到了最先进的性能。
• 跨数据库评估显示学习到的质量感知特征具有鲁棒的泛化能力。
• 分层特征融合（阶梯网络）更好地捕捉用于质量评估的低-中-高层信息。
• IMDT 通过在多个数据库上训练并使用数据库特定回归器，增加训练样本多样性并改善特征泛化。
• 在六个野外数据集上，所提方法明显优于竞争的 BIQA 模型。
• StairIQA 的代码已在项目仓库中发布。