[Paper Review] AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

Yuqin Cao, Xiongkuo Min+3 more|ArXiv.org|Jan 30, 2025

Noise Effects and Management4 citations

TL;DR

The paper introduces AGAV-Rater, an LMM-based model that scores AI-generated audio-visual content across multiple dimensions and selects the best AGAV, backed by the AGAVQA dataset.

ABSTRACT

Many video-to-audio (VTA) methods have been proposed for dubbing silent AI-generated videos. An efficient quality assessment method for AI-generated audio-visual content (AGAV) is crucial for ensuring audio-visual quality. Existing audio-visual quality assessment methods struggle with unique distortions in AGAVs, such as unrealistic and inconsistent elements. To address this, we introduce AGAVQA-3k, the first large-scale AGAV quality assessment dataset, comprising $3,382$ AGAVs from $16$ VTA methods. AGAVQA-3k includes two subsets: AGAVQA-MOS, which provides multi-dimensional scores for audio quality, content consistency, and overall quality, and AGAVQA-Pair, designed for optimal AGAV pair selection. We further propose AGAV-Rater, a LMM-based model that can score AGAVs, as well as audio and music generated from text, across multiple dimensions, and selects the best AGAV generated by VTA methods to present to the user. AGAV-Rater achieves state-of-the-art performance on AGAVQA-3k, Text-to-Audio, and Text-to-Music datasets. Subjective tests also confirm that AGAV-Rater enhances VTA performance and user experience. The dataset and code is available at https://github.com/charlotte9524/AGAV-Rater.

Motivation & Objective

• Motivate the need for automated AGAV quality assessment due to unique distortions in AI-generated audio-visual content (AGAV).
• Create AGAVQA, a large-scale AGAV quality assessment dataset with multi-dimensional MOS and optimal AGAV pair labels.
• Develop AGAV-Rater, an LMM-based model that can predict multi-dimensional scores for AGAVs, TTA, and TTM.
• Demonstrate AGAV-Rater's state-of-the-art performance and its usefulness in selecting higher-quality AGAV outputs.

Proposed method

• Construct AGAVQA with two subsets: AGAVQA-MOS for multi-dimensional MOS and AGAVQA-Pair for optimal AGAV pair selection.
• Pre-train AGAV-Rater using 50,952 instruction-response pairs derived from audio-caption datasets (VGGSound, AudioCaps, MusicCaps) with text-defined labels (excellent/bad).
• Fine-tune AGAV-Rater on human-annotated MOS across three dimensions (audio quality, content consistency, overall quality).
• Adopt a two-stage training: pre-training on text-defined levels, then fine-tuning via numerical scores (PLCC loss) on MOS data.
• Base AGAV-Rater on VideoLLaMA2 architecture, with separate video/audio encoders, projection to a shared space, and integration with text prompts for multi-modal evaluation.
• Evaluate using SRCC, KRCC, PLCC, RMSE across AGAVQA-MOS, TTA, and TTM datasets.

Experimental results

Research questions

• RQ1 Can LMMs be adapted to output numerical multi-dimensional quality scores for AGAV content?
• RQ2 Does a pre-trained AGAV-Rater generalize across unseen AGAV content and VTA methods?
• RQ3 Which training strategy (pretraining with text-defined levels vs. direct MOS finetuning) yields better multi-dimensional scoring accuracy?
• RQ4 Can AGAV-Rater improve selection of optimal AGAV outputs in real-world VTA applications?

Key findings

• AGAV-Rater achieves state-of-the-art performance on AGAVQA-MOS, Text-to-Audio (TTA), and Text-to-Music (TTM) datasets.
• Pretraining with text-defined levels substantially improves performance on smaller datasets (TTA and TTM).
• Multi-dimensional Instructions improve overall A/V quality prediction by leveraging audio quality and content consistency signals.
• AGAV-Rater demonstrates strong cross-dataset generalization, outperforming open- and closed-source LMMs on AGAVQA-Pair selection tasks.
• Subjective experiments show AGAV-Rater helps select higher-quality AGAVs, enhancing user experience (e.g., 80% preference in ElevenLabs enhancement test).