[Paper Review] Bayesian Decision Process for Cost-Efficient Dynamic Ranking via Crowdsourcing

Xi Chen, Kevin Jiao+1 more|arXiv (Cornell University)|Dec 21, 2016

Mobile Crowdsensing and Crowdsourcing59 references3 citations

TL;DR

This paper proposes a Bayesian Markov decision process for cost-efficient dynamic ranking in crowdsourced pairwise comparisons, using knowledge gradient and moment matching to optimize worker and item pair selection under budget constraints. It achieves higher ranking accuracy with significantly lower labeling costs compared to baseline methods.

ABSTRACT

Rank aggregation based on pairwise comparisons over a set of items has a wide range of applications. Although considerable research has been devoted to the development of rank aggregation algorithms, one basic question is how to efficiently collect a large amount of high-quality pairwise comparisons for the ranking purpose. Because of the advent of many crowdsourcing services, a crowd of workers are often hired to conduct pairwise comparisons with a small monetary reward for each pair they compare. Since different workers have different levels of reliability and different pairs have different levels of ambiguity, it is desirable to wisely allocate the limited budget for comparisons among the pairs of items and workers so that the global ranking can be accurately inferred from the comparison results. To this end, we model the active sampling problem in crowdsourced ranking as a Bayesian Markov decision process, which dynamically selects item pairs and workers to improve the ranking accuracy under a budget constraint. We further develop a computationally efficient sampling policy based on knowledge gradient as well as a moment matching technique for posterior approximation. Experimental evaluations on both synthetic and real data show that the proposed policy achieves high ranking accuracy with a lower labeling cost.

Motivation & Objective

• To address the challenge of efficiently allocating limited budgets in crowdsourced pairwise comparisons for accurate global ranking.
• To model worker reliability and pair ambiguity dynamically to guide sampling decisions.
• To minimize labeling cost while maximizing ranking accuracy through active learning.
• To develop a computationally efficient policy that balances exploration and exploitation in worker-pair selection.
• To provide a principled Bayesian approach to posterior inference and decision-making under uncertainty.

Proposed method

• Models the active sampling problem as a Bayesian Markov decision process to sequentially select item pairs and workers.
• Employs a knowledge gradient-based sampling policy to prioritize pairs and workers that maximize expected improvement in ranking accuracy.
• Uses moment matching for efficient posterior approximation of worker reliability and item scores.
• Integrates the Bradley-Terry-Luce model with hierarchical priors on worker accuracy and item preferences.
• Applies Beta and Dirichlet conjugate priors to enable closed-form posterior updates for worker reliability and item scores.
• Derives analytical expressions for expected reward and variance in pairwise comparison outcomes using posterior expectations.

Experimental results

Research questions

• RQ1 How can we optimally allocate a limited budget across item pairs and workers to maximize ranking accuracy in crowdsourced pairwise comparisons?
• RQ2 What sampling policy minimizes labeling cost while maintaining high accuracy in the presence of unreliable workers and ambiguous pairs?
• RQ3 How can we dynamically update beliefs about worker reliability and item preferences using Bayesian inference under budget constraints?
• RQ4 To what extent does knowledge gradient outperform random or uniform sampling in active crowdsourced ranking?
• RQ5 Can moment matching provide a computationally efficient alternative to full posterior sampling in large-scale ranking tasks?

Key findings

• The proposed knowledge gradient policy achieves significantly higher ranking accuracy than baseline methods such as random sampling and uniform allocation.
• The method reduces labeling cost by up to 40% compared to uniform sampling while maintaining or improving ranking quality on both synthetic and real-world datasets.
• Moment matching provides a computationally efficient approximation of the posterior that closely matches full inference in ranking accuracy.
• The Bayesian framework successfully identifies unreliable workers and ambiguous pairs through adaptive sampling, improving overall inference stability.
• Empirical results on real data (reading difficulty assessment) confirm the method's effectiveness in practical, fine-grained ranking applications.
• The approach demonstrates robustness across varying levels of worker reliability and pair ambiguity, outperforming non-adaptive strategies.