[Paper Review] Customized Video QoE Estimation with Algorithm-Agnostic Transfer Learning

Selim İckin, Markus Fiedler+1 more|arXiv (Cornell University)|Mar 12, 2020

Image and Video Quality Assessment15 references4 citations

TL;DR

This paper proposes an algorithm-agnostic transfer learning framework for customized video Quality of Experience (QoE) estimation that enables decentralized knowledge sharing without exposing sensitive local data. By training a generic base model on shared features and stacking it with a localized model trained on domain-specific features, the approach improves QoE prediction accuracy—achieving a 7% accuracy gain on small local datasets—while remaining agnostic to underlying machine learning algorithms and preserving privacy.

ABSTRACT

The development of QoE models by means of Machine Learning (ML) is challenging, amongst others due to small-size datasets, lack of diversity in user profiles in the source domain, and too much diversity in the target domains of QoE models. Furthermore, datasets can be hard to share between research entities, as the machine learning models and the collected user data from the user studies may be IPR- or GDPR-sensitive. This makes a decentralized learning-based framework appealing for sharing and aggregating learned knowledge in-between the local models that map the obtained metrics to the user QoE, such as Mean Opinion Scores (MOS). In this paper, we present a transfer learning-based ML model training approach, which allows decentralized local models to share generic indicators on MOS to learn a generic base model, and then customize the generic base model further using additional features that are unique to those specific localized (and potentially sensitive) QoE nodes. We show that the proposed approach is agnostic to specific ML algorithms, stacked upon each other, as it does not necessitate the collaborating localized nodes to run the same ML algorithm. Our reproducible results reveal the advantages of stacking various generic and specific models with corresponding weight factors. Moreover, we identify the optimal combination of algorithms and weight factors for the corresponding localized QoE nodes.

Motivation & Objective

• To address the challenge of training accurate QoE models in data-scarce, privacy-sensitive environments where local datasets are small and potentially GDPR/IPR-restricted.
• To enable decentralized knowledge sharing between localized QoE nodes without exposing sensitive user profiles or content-specific features.
• To decouple generic QoE factors (e.g., playback resolution, interruption) from specific ones (e.g., content type, spatio-temporal complexity) for improved model customization.
• To develop a transfer learning framework that is agnostic to the choice of machine learning algorithms at source and target domains.
• To demonstrate that stacking a pre-trained generic model with a localized model enhances prediction accuracy, especially when local data is limited.

Proposed method

• Decompose QoE modeling into two components: a generic base model trained on universally applicable features (e.g., resolution, interruptions) and a localized model trained on domain-specific features (e.g., content type, device context).
• Train the generic base model on a large, diverse source domain dataset to learn general QoE patterns, using features deemed universally representative.
• Train a separate localized model on the target domain’s data, including both generic and specific features, to capture context-dependent QoE effects.
• Use model stacking to combine predictions from the generic base model and the localized model, with learnable weight factors to optimize performance.
• Ensure algorithm-agnosticism by allowing different ML algorithms (e.g., XGBoost, Neural Networks) to be used at the source and target domains, enabling flexible and optimal model selection.
• Apply transfer learning by initializing the localized model with knowledge from the pre-trained generic base model, improving performance on small local datasets.

Experimental results

Research questions

• RQ1 Can a generic base model trained on diverse, large-scale data improve QoE prediction accuracy when fine-tuned with localized data in privacy-sensitive environments?
• RQ2 How does the performance of a stacked model compare to standalone local or generic models when the target domain has limited data?
• RQ3 To what extent does the choice of ML algorithm at the source and target domains affect the final stacked model performance, and can the framework support algorithm-agnostic model integration?
• RQ4 What is the optimal weight distribution between the generic base model and the localized model for maximizing prediction accuracy across different algorithm combinations?
• RQ5 How does the inclusion of content-specific features in the source domain affect transfer performance to a target domain with different content characteristics?

Key findings

• The generic base model trained on features like playback resolution and interruption performs with 0.75 accuracy on the source domain (G0), but drops to 0.71 when transferred to a smaller target domain (G1), indicating mild negative transfer.
• When the generic base model is trained with both generic and specific features (GF+SF), its performance drops more significantly on the target domain (0.68 vs. 0.80 on source), confirming that specific features in the source do not generalize well.
• Stacking the generic base model with a localized model trained on both generic and specific features improves accuracy on the small target dataset from 0.73 (standalone local) to 0.78 when using XGBoost at both ends.
• The optimal weight for the base model in the stacked ensemble is 0.5 when using the same algorithm (XGBoost) at source and target, indicating equal contribution from both models.
• When the base model is a Neural Network and the local model is XGBoost, the optimal base model weight is 0.3, suggesting the base model contributes less effectively in this configuration.
• The framework enables a 7% improvement in QoE estimation accuracy on small local datasets through privacy-preserving knowledge transfer, demonstrating its practical value in real-world deployment scenarios.