[Paper Review] Towards Deep Learning Methods for Quality Assessment of Computer-Generated Imagery

Markus Utke, Saman Zadtootaghaj+2 more|arXiv (Cornell University)|May 2, 2020

Image and Video Quality Assessment4 citations

TL;DR

This paper proposes a deep learning-based no-reference (NR) video quality assessment metric for computer-generated imagery (CGI), specifically targeting video gaming content. It leverages transfer learning by fine-tuning pre-trained Xception networks on frames annotated with VMAF scores, achieving high performance with only ~5,000 training frames, demonstrating that CNNs can effectively predict gaming video quality when trained on synthetic quality metrics.

ABSTRACT

Video gaming streaming services are growing rapidly due to new services such as passive video streaming, e.g. Twitch.tv, and cloud gaming, e.g. Nvidia Geforce Now. In contrast to traditional video content, gaming content has special characteristics such as extremely high motion for some games, special motion patterns, synthetic content and repetitive content, which makes the state-of-the-art video and image quality metrics perform weaker for this special computer generated content. In this paper, we outline our plan to build a deep learningbased quality metric for video gaming quality assessment. In addition, we present initial results by training the network based on VMAF values as a ground truth to give some insights on how to build a metric in future. The paper describes the method that is used to choose an appropriate Convolutional Neural Network architecture. Furthermore, we estimate the size of the required subjective quality dataset which achieves a sufficiently high performance. The results show that by taking around 5k images for training of the last six modules of Xception, we can obtain a relatively high performance metric to assess the quality of distorted video games.

Motivation & Objective

• To develop a deep learning-based no-reference (NR) video quality assessment metric tailored for computer-generated imagery (CGI), particularly gaming content.
• To investigate the feasibility of using pre-trained convolutional neural networks (CNNs) for quality assessment of gaming videos without access to reference signals.
• To determine the optimal CNN architecture, fine-tuning strategy, and minimum dataset size required for effective transfer learning in CGI quality assessment.
• To evaluate whether training on VMAF scores as a proxy for subjective quality can yield high-performing models before transitioning to real Mean Opinion Score (MOS) data.

Proposed method

• Fine-tune the last six modules of a pre-trained Xception CNN using a dataset of ~5,000 frames from video games, with VMAF scores as ground truth.
• Use transfer learning to adapt a pre-trained CNN architecture to the specific characteristics of computer-generated video content.
• Apply data sampling by selecting every n-th frame (n = 53) to reduce redundancy while maintaining model performance.
• Compare performance across different CNN architectures (e.g., VGG, Xception) and varying numbers of fine-tuned layers (1, 4, 6 modules).
• Evaluate model performance using RMSE and SRCC on both frame-level and video-level quality prediction.
• Investigate data augmentation strategies, including random cropping versus center cropping, to improve generalization.

Experimental results

Research questions

• RQ1 Are machine learning-based quality assessment methods suitable for computer-generated imagery?
• RQ2 How should a pre-trained deep CNN be retrained to achieve good performance on CGI quality assessment?
• RQ3 Which pre-trained CNN architecture performs best among state-of-the-art models for CGI quality assessment?
• RQ4 How much training data, in terms of number of frames, is required for effective transfer learning in this context?

Key findings

• Training the last six modules of the Xception network on approximately 5,000 frames yielded a high-performing metric with a video-level SRCC of 0.987 and RMSE of 3.11.
• The Xception architecture outperformed other models, achieving a video-level PCC of 0.987 and RMSE of 3.11 when fine-tuned on 5,287 frames (n=53).
• Increasing the number of fine-tuned modules improved performance, but with diminishing returns and increased overfitting risk on smaller datasets.
• Random cropping during training led to better performance than center cropping, suggesting greater data diversity improves generalization.
• Using every 53rd frame (n=53) provided a good balance between dataset size and model performance, with no significant drop in accuracy.
• Performance did not significantly improve when training and validation sets included overlapping games, indicating robustness to domain overlap in this setup.