[Paper Review] Space-Time Video Regularity and Visual Fidelity: Compression, Resolution and Frame Rate Adaptation

Dae Yeol Lee, Hyunsuk Ko+2 more|arXiv (Cornell University)|Mar 31, 2021

Image and Video Quality Assessment52 references4 citations

TL;DR

This paper proposes a novel video quality prediction model that leverages natural video statistics (NVS) of space-time displaced frame differences to predict perceptual quality under combined compression, resolution, and frame rate adaptation. By modeling divisively normalized frame differences along motion trajectories and using a learned regressor, the method achieves state-of-the-art performance on the ETRI-LIVE STSVQ database, demonstrating strong robustness to complex video distortions.

ABSTRACT

In order to be able to deliver today's voluminous amount of video contents through limited bandwidth channels in a perceptually optimal way, it is important to consider perceptual trade-offs of compression and space-time downsampling protocols. In this direction, we have studied and developed new models of natural video statistics (NVS), which are useful because high-quality videos contain statistical regularities that are disturbed by distortions. Specifically, we model the statistics of divisively normalized difference between neighboring frames that are relatively displaced. In an extensive empirical study, we found that those paths of space-time displaced frame differences that provide maximal regularity against our NVS model generally align best with motion trajectories. Motivated by this, we build a new video quality prediction engine that extracts NVS features from displaced frame differences, and combines them in a learned regressor that can accurately predict perceptual quality. As a stringent test of the new model, we apply it to the difficult problem of predicting the quality of videos subjected not only to compression, but also to downsampling in space and/or time. We show that the new quality model achieves state-of-the-art (SOTA) prediction performance compared on the new ETRI-LIVE Space-Time Subsampled Video Quality (STSVQ) database, which is dedicated to this problem. Downsampling protocols are of high interest to the streaming video industry, given rapid increases in frame resolutions and frame rates.

Motivation & Objective

• To address the challenge of predicting perceptual video quality under combined compression, spatial downsampling, and temporal downsampling.
• To model statistical regularities in natural video sequences that are disrupted by distortions.
• To develop a quality prediction engine that leverages space-time displaced frame differences for improved fidelity estimation.
• To evaluate the model on a new, challenging benchmark dataset specifically designed for space-time subsampling.
• To provide a perceptually accurate, learnable framework for video quality assessment in bandwidth-constrained streaming environments.

Proposed method

• The method models divisively normalized differences between temporally and spatially displaced video frames to capture space-time regularity.
• It identifies paths of displaced frame differences that maximize regularity under a learned natural video statistics (NVS) model.
• The model extracts NVS-based features from these displaced frame differences as input to a learned regressor.
• The regressor is trained to predict subjective video quality scores using a large-scale dataset with diverse distortions.
• The approach is validated on the ETRI-LIVE Space-Time Subsampled Video Quality (STSVQ) database, which includes combinations of compression, resolution, and frame rate changes.
• The model uses motion trajectory alignment as a proxy for perceptual quality, enhancing its sensitivity to realistic distortions.

Experimental results

Research questions

• RQ1 How can space-time displaced frame differences be used to model natural video statistics for quality prediction?
• RQ2 To what extent do paths of maximal regularity in displaced frame differences align with actual motion trajectories?
• RQ3 Can a learned regressor based on NVS features outperform existing models in predicting quality under combined compression, resolution, and frame rate adaptation?
• RQ4 How does the proposed model perform on a benchmark dataset specifically designed for space-time subsampling?
• RQ5 What is the relative contribution of spatial and temporal downsampling to perceptual video quality degradation?

Key findings

• The proposed model achieves state-of-the-art prediction performance on the ETRI-LIVE STSVQ database, outperforming existing methods in predicting quality under complex, real-world distortions.
• Paths of displaced frame differences that maximize regularity under the NVS model show strong alignment with actual motion trajectories, validating the model's perceptual relevance.
• The use of divisively normalized frame differences significantly improves the model's sensitivity to perceptual distortions compared to standard difference-based features.
• The learned regressor effectively generalizes across diverse combinations of compression, spatial downsampling, and temporal downsampling.
• The model demonstrates robustness to high levels of distortion, maintaining high prediction accuracy even under extreme bandwidth constraints.
• The results confirm that space-time regularity is a strong predictor of visual fidelity, especially when combined with motion-aware feature extraction.