[Paper Review] Bayesian Paired-Comparison with the bpcs Package

David Issa Mattos, Érika Martins Silva Ramos|arXiv (Cornell University)|Jan 27, 2021

Statistical Methods and Bayesian Inference49 references4 citations

TL;DR

This paper introduces the bpcs R package, which implements Bayesian paired comparison models using Stan for robust, flexible analysis of paired comparison data in behavioral research. It enables estimation under non-identifiable likelihoods, provides credible intervals, handles ties and order effects, and offers posterior inference on ranks and pairwise probabilities—demonstrated through reanalyses of three studies showing improved model stability and interpretability over frequentist approaches.

ABSTRACT

This article introduces the bpcs R package (Bayesian Paired Comparison in Stan) and the statistical models implemented in the package. This package aims to facilitate the use of Bayesian models for paired comparison data in behavioral research. Bayesian analysis of paired comparison data allows parameter estimation even in conditions where the maximum likelihood does not exist, allows easy extension of paired comparison models, provide straightforward interpretation of the results with credible intervals, have better control of type I error, have more robust evidence towards the null hypothesis, allows propagation of uncertainties, includes prior information, and perform well when handling models with many parameters and latent variables. The bpcs package provides a consistent interface for R users and several functions to evaluate the posterior distribution of all parameters, to estimate the posterior distribution of any contest between items, and to obtain the posterior distribution of the ranks. Three reanalyses of recent studies that used the frequentist Bradley-Terry model are presented. These reanalyses are conducted with the Bayesian models of the bpcs package, and all the code used to fit the models, generate the figures, and the tables are available in the online appendix.

Motivation & Objective

• To address limitations in frequentist paired comparison models, such as non-identifiable maximum likelihood estimates and poor control of type I error.
• To provide a comprehensive, user-friendly Bayesian alternative to the frequentist Bradley-Terry model and its extensions.
• To enable researchers to incorporate prior knowledge, handle latent variables, and propagate uncertainty in parameter estimates.
• To offer tools for posterior inference on pairwise comparisons, ranks, and model predictions with credible intervals.
• To demonstrate the advantages of Bayesian modeling through reanalyses of three real-world behavioral science studies.

Proposed method

• Uses Stan for full Bayesian inference via Markov Chain Monte Carlo (MCMC) sampling to estimate posterior distributions of parameters.
• Implements the Bayesian Bradley-Terry model and its extensions, including the Davidson model for tied outcomes and models with order effects.
• Supports hierarchical modeling with subject-specific predictors (e.g., HLOC dimensions) and generalized linear models for item-level covariates.
• Provides functions to compute posterior probabilities of item rankings and pairwise comparisons between items.
• Uses highest posterior density (HPD) intervals for credible interval estimation and model diagnostics.
• Offers a consistent R interface with functions for model fitting, posterior summarization, and visualization of results.

Experimental results

Research questions

• RQ1 How can Bayesian modeling improve parameter estimation in paired comparison data when maximum likelihood estimation fails?
• RQ2 What advantages does the Bayesian approach offer in terms of type I error control and evidence toward the null hypothesis?
• RQ3 How do subject-specific predictors such as HLOC dimensions affect the probability of selecting one role over another in role preference tasks?
• RQ4 To what extent do model extensions like ties and order effects improve fit and interpretation in real-world behavioral data?
• RQ5 How does the posterior distribution of ranks and pairwise comparisons compare to frequentist point estimates in terms of uncertainty and robustness?

Key findings

• The Bayesian model successfully estimated parameters even in cases where maximum likelihood estimation failed, demonstrating robustness in sparse or unbalanced data.
• For the role preference study, a subject with internal HLOC two standard deviations below the mean had a 0.80 probability of choosing 'active' over 'passive', compared to 0.88 for the average, indicating a small but measurable effect.
• The probability of choosing 'active' over 'passive' decreased to 0.78 when the 'doctors' HLOC dimension was two standard deviations above the mean, showing a modest but significant influence.
• The posterior probability of selecting 'collaborative' over 'passive-collaborative' was 0.98 for average subjects, dropping to 0.94 when 'chance' HLOC was two SD above the mean.
• The model with subject-specific predictors improved fit, but the baseline role coefficients contributed more to differences in selection probabilities than the predictors themselves.
• All reanalyses demonstrated that Bayesian models provide clearer, more interpretable uncertainty estimates through credible intervals and posterior predictive distributions.