[Paper Review] HVS Revisited: A Comprehensive Video Quality Assessment Framework

Aoxiang Zhang, Yuan‐Gen Wang+3 more|arXiv (Cornell University)|Oct 9, 2022

Image and Video Quality Assessment6 citations

TL;DR

This paper proposes HVS-5M, a no-reference video quality assessment framework that revisits the human visual system (HVS) by integrating five key characteristics—visual saliency, content-dependency, edge masking, motion perception, and temporal hysteresis—into a domain-fusion architecture. The framework uses SAMNet, ConvNeXt, SlowFast, and TempHyst to extract and fuse spatial and temporal features, achieving state-of-the-art performance across multiple datasets with superior generalization and robustness.

ABSTRACT

Video quality is a primary concern for video service providers. In recent years, the techniques of video quality assessment (VQA) based on deep convolutional neural networks (CNNs) have been developed rapidly. Although existing works attempt to introduce the knowledge of the human visual system (HVS) into VQA, there still exhibit limitations that prevent the full exploitation of HVS, including an incomplete model by few characteristics and insufficient connections among these characteristics. To overcome these limitations, this paper revisits HVS with five representative characteristics, and further reorganizes their connections. Based on the revisited HVS, a no-reference VQA framework called HVS-5M (NRVQA framework with five modules simulating HVS with five characteristics) is proposed. It works in a domain-fusion design paradigm with advanced network structures. On the side of the spatial domain, the visual saliency module applies SAMNet to obtain a saliency map. And then, the content-dependency and the edge masking modules respectively utilize ConvNeXt to extract the spatial features, which have been attentively weighted by the saliency map for the purpose of highlighting those regions that human beings may be interested in. On the other side of the temporal domain, to supplement the static spatial features, the motion perception module utilizes SlowFast to obtain the dynamic temporal features. Besides, the temporal hysteresis module applies TempHyst to simulate the memory mechanism of human beings, and comprehensively evaluates the quality score according to the fusion features from the spatial and temporal domains. Extensive experiments show that our HVS-5M outperforms the state-of-the-art VQA methods. Ablation studies are further conducted to verify the effectiveness of each module towards the proposed framework.

Motivation & Objective

• To address the limitations of existing video quality assessment (VQA) methods that incompletely model the human visual system (HVS) and poorly connect its characteristics.
• To improve performance on in-the-wild videos by integrating underexplored HVS features such as edge masking and enhanced visual saliency.
• To develop a robust, generalizable VQA framework that effectively fuses spatial and temporal features using a domain-fusion design paradigm.
• To validate the effectiveness of each component through ablation studies and demonstrate the framework's superiority over state-of-the-art methods.

Proposed method

• The HVS-5M framework is built on a revisited HVS model incorporating five characteristics: visual saliency, content-dependency, edge masking, motion perception, and temporal hysteresis.
• In the spatial domain, visual saliency is modeled using SAMNet to generate a saliency map, which attentionally weights features from ConvNeXt-based content-dependency and edge masking modules.
• In the temporal domain, motion perception is captured via the SlowFast network to extract dynamic features, while temporal hysteresis is modeled using the TempHyst module to simulate human memory effects.
• Spatial and temporal features are fused and processed through a fully connected layer to predict the final video quality score.
• The framework employs a domain-fusion design, enabling joint optimization of spatial and temporal representations for improved quality estimation.
• Advanced architectures like ConvNeXt and TempHyst are integrated to enhance feature representation and temporal modeling.

Experimental results

Research questions

• RQ1 How can the human visual system (HVS) be more comprehensively modeled in video quality assessment by integrating multiple interrelated characteristics?
• RQ2 What is the contribution of edge masking and visual saliency to video quality prediction in the absence of pristine references?
• RQ3 How does the integration of temporal hysteresis improve the robustness and accuracy of no-reference VQA models?
• RQ4 To what extent does the domain-fusion design of spatial and temporal features enhance generalization across diverse video datasets?

Key findings

• HVS-5M achieves state-of-the-art performance on five mainstream video quality assessment datasets, including LIVE-VQC, KoNViD-1k, and CVD2014.
• Ablation studies show that disabling the visual saliency module leads to performance degradation on four of the six datasets, confirming its critical role in attention-based feature weighting.
• The edge masking module contributes significantly, with a 2.5% average SRCC degradation when disabled, indicating its importance in high-contrast or textured regions.
• Replacing the TempHyst module with a fully connected network causes a 16.6% SRCC drop on KoNViD-1k, proving that temporal memory modeling is essential for accurate quality prediction.
• Using ResNet-50 instead of ConvNeXt in key modules results in a 3.5% average SRCC degradation, demonstrating the superiority of ConvNeXt for feature extraction in this framework.
• The framework shows strong generalization across categorical, cross-dataset, and mixed-dataset evaluations, indicating robustness to distribution shifts.