[Paper Review] Generalized Score Distribution

Lucjan Janowski, Bogdan Ćmiel+3 more|arXiv (Cornell University)|Sep 10, 2019

Advanced Statistical Methods and Models28 references4 citations

TL;DR

This paper proposes the Generalized Score Distribution (GSD), a two-parameter discrete probability distribution for finite-support data like Likert-scale responses in subjective quality assessments. It models both the mean (ψ) and spread (ρ) of answers, demonstrating with 99.7% probability that real video quality scores from four databases follow GSD—outperforming normal and quasi-normal models—making it ideal for accurate data analysis and simulation in quality-of-experience research.

ABSTRACT

A class of discrete probability distributions contains distributions with limited support, i.e. possible argument values are limited to a set of numbers (typically consecutive). Examples of such data are results from subjective experiments utilizing the Absolute Category Rating (ACR) technique, where possible answers (argument values) are $\{1, 2, \cdots, 5\}$ or typical Likert scale $\{-3, -2, \cdots, 3\}$. An interesting subclass of those distributions are distributions limited to two parameters: describing the mean value and the spread of the answers, and having no more than one change in the probability monotonicity. In this paper we propose a general distribution passing those limitations called Generalized Score Distribution (GSD). The proposed GSD covers all spreads of the answers, from very small, given by the Bernoulli distribution, to the maximum given by a Beta Binomial distribution. We also show that GSD correctly describes subjective experiments scores from video quality evaluations with probability of 99.7\%. A Google Collaboratory website with implementation of the GSD estimation, simulation, and visualization is provided.

Motivation & Objective

• To develop a general, discrete probability distribution that accurately models subjective responses on finite scales, such as 5-point Likert scales used in video quality testing.
• To address limitations in existing models that rely on continuous approximations with discretization and censoring, which introduce modeling errors.
• To provide a robust statistical foundation for analyzing and simulating subjective experiment data, particularly in quality-of-experience (QoE) research.
• To validate the proposed GSD model using real-world databases and demonstrate its superiority over conventional models like the normal and quasi-normal distributions.

Proposed method

• Propose a new discrete distribution, GSD, defined over a finite support {1, 2, ..., M}, with two parameters: ψ (mean) and ρ (spread), ensuring at most one change in probability monotonicity.
• Derive the probability mass function of GSD using a generalized form that includes the Bernoulli (ρ → 0) and Beta-Binomial (ρ → 1) distributions as limiting cases.
• Implement a parameter estimation algorithm using maximum likelihood, validated through extensive simulation studies.
• Apply the GSD model to 1,874 processed video sequences (PVSs) from publicly available databases, comparing goodness-of-fit with normal and quasi-normal models using χ² tests.
• Use a Google Colab-based implementation for GSD estimation, simulation, and visualization, enabling reproducibility and practical use.
• Conduct a formal goodness-of-fit test pipeline (Fig. 11) comparing observed frequencies to theoretical probabilities under GSD, QNormal, and Normal models.

Experimental results

Research questions

• RQ1 Does the Generalized Score Distribution (GSD) provide a better fit to subjective video quality scores than traditional normal or quasi-normal models?
• RQ2 Can GSD accurately model the full range of answer spread, from low variability (Bernoulli-like) to high variability (Beta-Binomial-like), within a finite discrete scale?
• RQ3 What are the typical parameter distributions of ψ and ρ across real-world video quality databases, and can they be used as informative priors in Bayesian analysis?
• RQ4 Is the GSD model robust and generalizable beyond video quality assessment, particularly for other Likert-scale subjective experiments?

Key findings

• The GSD model fits subjective video quality scores with a probability of 99.7%, as confirmed by a χ² goodness-of-fit test across four databases containing 1,874 processed video sequences.
• The p-value for GSD goodness-of-fit was 0.997, indicating strong evidence that the data follow the GSD distribution, whereas p-values for QNormal and Normal models were 0.000005612 and 0.000000000, respectively, indicating poor fit.
• The distribution of the ψ parameter (mean) across PVSs showed a lack of very high scores (close to 5), suggesting potential database bias, while the ρ parameter (spread) was well-approximated by a truncated normal distribution with µρ = 0.86 and σρ = 0.071.
• The ρ parameter effectively characterizes the underlying response process: values near 1 indicate high variability (Beta-Binomial-like), while values near 0 indicate low variability (Bernoulli-like), enabling diagnostic use.
• The GSD model outperforms both the normal and quasi-normal models in fitting real subjective data, with the latter rejecting the null hypothesis of fit at the 5% significance level.
• The authors provide a publicly available Google Colab implementation for GSD estimation, simulation, and visualization, supporting reproducibility and practical adoption.