[Paper Review] UNQA: Unified No-Reference Quality Assessment for Audio, Image, Video, and Audio-Visual Content

Yuqin Cao, Xiongkuo Min+4 more|arXiv (Cornell University)|Jul 29, 2024

Infrastructure Maintenance and Monitoring4 citations

TL;DR

This paper proposes UNQA, the first unified no-reference quality assessment model that jointly predicts perceptual quality for audio, image, video, and audio-visual content using a single architecture. By employing a multi-modality training strategy to align disparate quality scales across databases, UNQA achieves state-of-the-art performance with reduced model complexity and superior generalization across diverse media types.

ABSTRACT

As multimedia data flourishes on the Internet, quality assessment (QA) of multimedia data becomes paramount for digital media applications. Since multimedia data includes multiple modalities including audio, image, video, and audio-visual (A/V) content, researchers have developed a range of QA methods to evaluate the quality of different modality data. While they exclusively focus on addressing the single modality QA issues, a unified QA model that can handle diverse media across multiple modalities is still missing, whereas the latter can better resemble human perception behaviour and also have a wider range of applications. In this paper, we propose the Unified No-reference Quality Assessment model (UNQA) for audio, image, video, and A/V content, which tries to train a single QA model across different media modalities. To tackle the issue of inconsistent quality scales among different QA databases, we develop a multi-modality strategy to jointly train UNQA on multiple QA databases. Based on the input modality, UNQA selectively extracts the spatial features, motion features, and audio features, and calculates a final quality score via the four corresponding modality regression modules. Compared with existing QA methods, UNQA has two advantages: 1) the multi-modality training strategy makes the QA model learn more general and robust quality-aware feature representation as evidenced by the superior performance of UNQA compared to state-of-the-art QA methods. 2) UNQA reduces the number of models required to assess multimedia data across different modalities. and is friendly to deploy to practical applications.

Motivation & Objective

• To address the lack of a unified quality assessment model that can evaluate multiple media modalities simultaneously.
• To overcome the challenge of inconsistent perceptual quality scales across different QA databases (e.g., [0,5] vs. [0,100]).
• To develop a single, parameter-efficient model that generalizes across audio, image, video, and audio-visual content.
• To improve robustness and deployability in real-world applications where media may be incomplete or multimodal.

Proposed method

• UNQA uses a shared backbone with modality-specific feature extraction modules: spatial for images, motion for video, and audio for audio content.
• It employs multi-head self-attention in the spatial feature extractor to emphasize perceptually salient regions.
• A novel multi-modality training strategy pre-trains feature extractors on database-specific regression heads and then fine-tunes the full model using relative ranking loss instead of raw MOS values.
• The model fuses modality-specific features through four dedicated regression heads to predict quality scores for each modality or a combined A/V score.
• Grad-CAM is used to visualize attention maps, demonstrating that UNQA focuses on perceptually relevant regions such as people, text, and structures.
• Computational efficiency is evaluated on a standard GPU, showing UNQA has lower inference time than competing deep learning models.

Experimental results

Research questions

• RQ1 Can a single unified model achieve high-quality no-reference assessment across audio, image, video, and audio-visual content?
• RQ2 How can disparate perceptual quality scales across multiple QA databases be effectively aligned during joint training?
• RQ3 Does joint training on multiple modalities lead to more robust and generalizable quality-aware representations?
• RQ4 Can a unified model outperform modality-specific state-of-the-art models while reducing model complexity?

Key findings

• UNQA achieves state-of-the-art performance across all four modalities, with a Spearman’s rank correlation coefficient (SRCC) of 0.8037 on image QA, 0.8901 on video, 0.8791 on audio-visual, and 0.7962 on audio.
• In cross-database evaluation, UNQA generalizes well to unseen databases, outperforming 20 SOTA methods on seven external QA datasets.
• UNQA demonstrates superior computational efficiency, with the lowest running time among all compared models on a standard 1080p, 25fps video.
• Visual analysis using Grad-CAM confirms that UNQA focuses on perceptually salient regions such as faces, text, and buildings, aligning with human perception.
• The multi-modality training strategy effectively mitigates scale discrepancies between databases without requiring re-sampling or reweighting of MOS values.
• UNQA reduces the number of required models from one per modality to a single unified model, enhancing deployability in real-world systems.