[Paper Review] Perceived Audiovisual Quality Modelling based on Decison Trees, Genetic Programming and Neural Networks

Edip Demirbilek, Jean‐Charles Grégoire|arXiv (Cornell University)|Dec 6, 2017

Image and Video Quality Assessment21 references3 citations

TL;DR

This paper proposes machine learning models—specifically Random Forests, Bagging, Deep Learning, and Genetic Programming—that predict audiovisual quality directly from bitstream features extracted from a real-time multimedia dataset. Using the INRS audiovisual quality dataset with 125 correlated features, Random Forests and Bagging achieved the highest accuracy (lowest RMSE and highest Pearson correlation), outperforming Deep Learning and Genetic Programming, which required fewer features to reach peak performance.

ABSTRACT

Our objective is to build machine learning based models that predict audiovisual quality directly from a set of correlated parameters that are extracted from a target quality dataset. We have used the bitstream version of the INRS audiovisual quality dataset that reflects contemporary real-time configurations for video frame rate, video quantization, noise reduction parameters and network packet loss rate. We have utilized this dataset to build bitstream perceived quality estimation models based on the Random Forests, Bagging, Deep Learning and Genetic Programming methods. We have taken an empirical approach and have generated models varying from very simple to the most complex depending on the number of features used from the quality dataset. Random Forests and Bagging models have overall generated the most accurate results in terms of RMSE and Pearson correlation coefficient values. Deep Learning and Genetic Programming based bitstream models have also achieved good results but that high performance was observed only with a limited range of features. We have also obtained the epsilon-insensitive RMSE values for each model and have computed the significance of the difference between the correlation coefficients. Overall we conclude that computing the bitstream information is worth the effort it takes to generate and helps to build more accurate models for real-time communications. However, it is useful only for the deployment of the right algorithms with the carefully selected subset of the features. The dataset and tools that have been developed during this research are publicly available for research and development purposes.

Motivation & Objective

• To develop machine learning models that predict perceived audiovisual quality directly from bitstream parameters without intermediate quality functions.
• To evaluate the performance of diverse machine learning algorithms—Random Forests, Bagging, Deep Learning, and Genetic Programming—on a real-time multimedia quality dataset.
• To determine the optimal subset of bitstream features that maximize prediction accuracy for real-time communication systems.
• To assess the trade-offs between model complexity, training effort, and prediction accuracy in perceived quality modeling.
• To make the dataset and tools publicly available for future research and development in quality of experience (QoE) modeling.

Proposed method

• Constructed a bitstream version of the INRS audiovisual quality dataset containing 125 features derived from media and network parameters such as video frame rate, quantization, noise reduction, and packet loss rate.
• Used leave-one-out cross-validation with Random Forests to rank feature importance and generate 125 incremental models per algorithm (1 to 125 features).
• Trained and evaluated four machine learning models: Random Forests, Bagging, Deep Learning, and Genetic Programming on the same dataset using increasing feature subsets.
• Measured model performance using RMSE, Pearson correlation coefficient, and epsilon-insensitive RMSE (RMSE*) to assess robustness and significance of differences.
• Applied Fisher-z transformation to test the statistical significance of differences between correlation coefficients across models.
• Selected optimal feature subsets based on performance trends and computational cost, emphasizing model efficiency and accuracy trade-offs.

Experimental results

Research questions

• RQ1 Which machine learning algorithm achieves the highest accuracy in predicting audiovisual quality from bitstream features?
• RQ2 How does model performance vary with the number of input features, and what is the optimal feature subset size for each algorithm?
• RQ3 What is the trade-off between model complexity, training time, and prediction accuracy across different algorithms?
• RQ4 How do the performance metrics (RMSE, correlation, RMSE*) compare across Random Forests, Bagging, Deep Learning, and Genetic Programming?
• RQ5 To what extent does computing bitstream information improve the accuracy of perceived quality models in real-time communication systems?

Key findings

• Random Forests and Bagging models achieved the best overall performance, with the lowest RMSE and highest Pearson correlation coefficient across all feature counts.
• Deep Learning and Genetic Programming models reached peak performance with only the top 7 features, indicating high sensitivity to feature selection.
• Random Forests-based models outperformed all others, especially when using the top 20 features, and required significantly less tuning and training time.
• Genetic Programming models required the most computational resources, longest training time, and extensive hyperparameter tuning, yet delivered the lowest performance.
• RMSE* values confirmed that model performance differences became more pronounced as feature count increased beyond two features, indicating added value in feature richness.
• The study confirms that extracting bitstream features is worthwhile for improving model accuracy, but only when paired with the right algorithm and carefully selected feature subset.