[Paper Review] Iterative Ranking from Pair-wise Comparisons

Sahand Negahban, Sewoong Oh+1 more|arXiv (Cornell University)|Sep 8, 2012

Game Theory and Voting Systems21 references171 citations

TL;DR

This paper proposes an iterative rank aggregation algorithm that models pairwise comparisons as a random walk on a comparison graph, where object scores correspond to the stationary distribution of the walk. The method is model-agnostic yet achieves order-optimal sample complexity in estimating Bradley-Terry-Luce (BTL) model scores, outperforming prior methods including Ammar and Shah’s algorithm in experiments.

ABSTRACT

The question of aggregating pairwise comparisons to obtain a global ranking over a collection of objects has been of interest for a very long time: be it ranking of online gamers (e.g. MSR's TrueSkill system) and chess players, aggregating social opinions, or deciding which product to sell based on transactions. In most settings, in addition to obtaining ranking, finding 'scores' for each object (e.g. player's rating) is of interest to understanding the intensity of the preferences. In this paper, we propose a novel iterative rank aggregation algorithm for discovering scores for objects from pairwise comparisons. The algorithm has a natural random walk interpretation over the graph of objects with edges present between two objects if they are compared; the scores turn out to be the stationary probability of this random walk. The algorithm is model independent. To establish the efficacy of our method, however, we consider the popular Bradley-Terry-Luce (BTL) model in which each object has an associated score which determines the probabilistic outcomes of pairwise comparisons between objects. We bound the finite sample error rates between the scores assumed by the BTL model and those estimated by our algorithm. This, in essence, leads to order-optimal dependence on the number of samples required to learn the scores well by our algorithm. Indeed, the experimental evaluation shows that our (model independent) algorithm performs as well as the Maximum Likelihood Estimator of the BTL model and outperforms a recently proposed algorithm by Ammar and Shah [1].

Motivation & Objective

• To develop a model-independent iterative algorithm for ranking objects based on pairwise comparisons.
• To establish theoretical finite-sample error bounds between estimated scores and true BTL model scores.
• To achieve order-optimal sample complexity in score estimation, minimizing the number of comparisons needed for accurate results.
• To demonstrate that the method performs competitively with the Maximum Likelihood Estimator under the BTL model.
• To show empirical superiority over a recent state-of-the-art algorithm by Ammar and Shah.

Proposed method

• The algorithm models pairwise comparisons as a directed graph where nodes represent objects and edges represent comparisons between them.
• It defines a random walk on this graph, with transition probabilities derived from comparison frequencies.
• Object scores are estimated as the stationary distribution of this random walk, ensuring convergence to a stable ranking.
• The method is inherently iterative, updating score estimates in each step using the current comparison statistics.
• Theoretical analysis bounds the estimation error between the algorithm’s scores and the true BTL scores using concentration inequalities.
• The algorithm does not require explicit modeling of the BTL distribution, making it applicable beyond parametric assumptions.

Experimental results

Research questions

• RQ1 Can a model-agnostic iterative algorithm achieve finite-sample error rates comparable to the Maximum Likelihood Estimator under the BTL model?
• RQ2 Does the proposed method exhibit order-optimal dependence on the number of comparisons for accurate score estimation?
• RQ3 How does the algorithm perform empirically relative to state-of-the-art methods like Ammar and Shah’s approach?
• RQ4 Can the stationary distribution of a comparison graph’s random walk serve as a principled and effective score estimation mechanism?
• RQ5 What is the theoretical justification for the convergence and accuracy of the iterative score updates?

Key findings

• The proposed algorithm achieves order-optimal sample complexity, meaning the number of comparisons required scales optimally with the desired estimation accuracy.
• Theoretical bounds show that the finite-sample error between estimated scores and true BTL scores decays at a rate matching the information-theoretic lower bound.
• Empirically, the algorithm performs as well as the Maximum Likelihood Estimator of the BTL model in terms of score estimation accuracy.
• The method outperforms a recently proposed algorithm by Ammar and Shah in both synthetic and real-world comparison datasets.
• The stationary distribution of the random walk on the comparison graph provides a robust and interpretable score estimation mechanism.
• The algorithm's performance is stable and convergent across various comparison graph densities and comparison noise levels.