[Paper Review] Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception

Bowen Li, Weixia Zhang+3 more|arXiv (Cornell University)|Aug 19, 2021

Image and Video Quality Assessment4 citations

TL;DR

This paper proposes a blind video quality assessment (BVQA) method that leverages quality-aware pre-training on image quality assessment (IQA) databases and motion perception from action recognition models to learn complementary spatio-temporal features. By combining these representations with a mixed list-wise ranking loss, the method achieves state-of-the-art performance on six in-the-wild VQA benchmarks, demonstrating strong generalization and efficiency, especially when accelerated via GPU.

ABSTRACT

Perceptual quality assessment of the videos acquired in the wilds is of vital importance for quality assurance of video services. The inaccessibility of reference videos with pristine quality and the complexity of authentic distortions pose great challenges for this kind of blind video quality assessment (BVQA) task. Although model-based transfer learning is an effective and efficient paradigm for the BVQA task, it remains to be a challenge to explore what and how to bridge the domain shifts for better video representation. In this work, we propose to transfer knowledge from image quality assessment (IQA) databases with authentic distortions and large-scale action recognition with rich motion patterns. We rely on both groups of data to learn the feature extractor. We train the proposed model on the target VQA databases using a mixed list-wise ranking loss function. Extensive experiments on six databases demonstrate that our method performs very competitively under both individual database and mixed database training settings. We also verify the rationality of each component of the proposed method and explore a simple manner for further improvement.

Motivation & Objective

• To address the challenge of blind video quality assessment in real-world scenarios where pristine references are unavailable and distortions are diverse and authentic.
• To reduce domain shift between source (e.g., image classification) and target (in-the-wild VQA) domains by using quality-aware pre-training on IQA databases.
• To enhance video representation by integrating motion perception from pre-trained 3D action recognition models.
• To improve generalization and performance through a mixed list-wise ranking loss function that jointly optimizes PLCC and SRCC.
• To achieve high efficiency and scalability for real-world deployment, especially at high video resolutions.

Proposed method

• Pre-train a frame-level feature extractor on multiple IQA databases with human-annotated quality scores to learn perceptually meaningful spatial features.
• Utilize a pre-trained 3D convolutional neural network (e.g., I3D) on action recognition datasets to extract motion-aware spatio-temporal features.
• Fusing the quality-aware spatial features and motion features through concatenation or attention mechanisms to form a comprehensive video representation.
• Train the entire BVQA model using a mixed list-wise ranking loss that combines Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank Correlation (SRCC) objectives.
• Apply model-based transfer learning to bridge domain shifts between source (IQA and action recognition) and target (in-the-wild VQA) domains.
• Optimize the model with end-to-end fine-tuning on target VQA databases, enabling effective adaptation to authentic distortions.

Experimental results

Research questions

• RQ1 Can quality-aware pre-training on IQA databases improve feature representation for blind video quality assessment in the wild?
• RQ2 Does incorporating motion perception from action recognition models enhance the performance of BVQA models on authentic distortions?
• RQ3 How effective is a mixed list-wise ranking loss in improving correlation with human opinion scores across diverse VQA databases?
• RQ4 To what extent does the proposed method generalize across multiple in-the-wild VQA databases under both individual and mixed training settings?
• RQ5 How does the computational efficiency of the proposed method scale with video resolution, especially when using GPU acceleration?

Key findings

• The proposed method achieves state-of-the-art performance on six benchmark in-the-wild VQA databases, with consistently high SRCC and PLCC values across all datasets.
• The model demonstrates strong generalization, performing competitively under both individual-database and mixed-database training settings.
• The integration of quality-aware pre-training and motion perception leads to a significant performance gain compared to models using only spatial features or standard ImageNet pre-training.
• The mixed list-wise ranking loss function further improves performance by jointly optimizing for both linear and rank-order correlation with human opinion scores.
• GPU-accelerated inference achieves up to 16× speedup over TLVQM at 2160p resolution, showing strong scalability with resolution.
• The method maintains high accuracy even on CPU, though GPU acceleration delivers a substantial efficiency advantage, especially at higher resolutions.