[Paper Review] Maximum Selection and Ranking under Noisy Comparisons

Moein Falahatgar, Alon Orlitsky+2 more|arXiv (Cornell University)|May 15, 2017

Game Theory and Voting SystemsEconomics, Econometrics and Finance8 references18 citations

TL;DR

This paper proposes an optimized knockout tournament algorithm for maximum selection under noisy comparisons, achieving ${\cal O}\left(\frac{n}{\epsilon^{2}}\log\frac{1}{\delta}\right)$ comparisons—optimal up to a constant factor—under strong stochastic transitivity and stochastic triangle inequality. It further introduces a merge-sort-based ranking framework that achieves ${\cal O}\left(\frac{n\log n(\log\log n)^3}{\epsilon^2}\right)$ comparisons for $\delta \geq \frac{1}{n}$, optimal up to a $(\log\log n)^3$ factor.

ABSTRACT

We consider $(ε,δ)$-PAC maximum-selection and ranking for general probabilistic models whose comparisons probabilities satisfy strong stochastic transitivity and stochastic triangle inequality. Modifying the popular knockout tournament, we propose a maximum-selection algorithm that uses $\mathcal{O}\left(\frac{n}{ε^2}\log \frac{1}δ ight)$ comparisons, a number tight up to a constant factor. We then derive a general framework that improves the performance of many ranking algorithms, and combine it with merge sort and binary search to obtain a ranking algorithm that uses $\mathcal{O}\left(\frac{n\log n (\log \log n)^3}{ε^2} ight)$ comparisons for any $δ\ge\frac1n$, a number optimal up to a $(\log \log n)^3$ factor.

Motivation & Objective

• To address the complexity gap in maximum selection and ranking under noisy comparisons, particularly when comparison outcomes are probabilistic.
• To develop algorithms that achieve near-optimal sample complexity under general probabilistic models satisfying strong stochastic transitivity and stochastic triangle inequality.
• To close the gap in known bounds for ranking, which previously had a $\log n$-factor uncertainty, by introducing a framework that improves existing algorithms.
• To provide theoretical guarantees for $(\epsilon,\delta)$-PAC maximum selection and ranking, ensuring high-probability correctness with minimal comparisons.

Proposed method

• Modifies the knockout tournament algorithm to achieve optimal $O\left(\frac{n}{\epsilon^2} \log \frac{1}{\delta}\right)$ comparisons for maximum selection under noisy comparisons.
• Introduces a general framework combining merge sort and binary search to improve ranking algorithm performance.
• Employs a recursive binary search strategy over candidate sets to refine rankings with bounded error $\epsilon$.
• Uses concentration inequalities and union bounds to ensure high-probability correctness across all comparisons.
• Applies Fano’s inequality in a proof sketch to establish a lower bound on sample complexity, demonstrating optimality up to logarithmic factors.
• Implements and evaluates variants of existing algorithms (e.g., IR-BTM-PAC) to validate the superiority of the proposed methods.

Experimental results

Research questions

• RQ1 Can maximum selection be achieved with sample complexity optimal up to a constant factor under noisy comparisons satisfying strong stochastic transitivity and triangle inequality?
• RQ2 What is the optimal sample complexity for ranking under the same probabilistic constraints, and can it be improved beyond existing $\log n$-factor gaps?
• RQ3 Can a general framework be designed to enhance the performance of multiple ranking algorithms under noisy comparison models?
• RQ4 How does the proposed merge-sort-based ranking algorithm compare in sample complexity to existing PAC ranking methods like BTM-PAC or MallowsMPI?
• RQ5 Is the proposed Knockout algorithm robust and efficient even when $\epsilon$ is small, particularly in cases with large pairwise preference gaps?

Key findings

• The proposed maximum selection algorithm achieves $O\left(\frac{n}{\epsilon^2} \log \frac{1}{\delta}\right)$ comparisons, which is optimal up to a constant factor under the given model assumptions.
• The ranking algorithm using the merge-sort and binary search framework achieves $O\left(\frac{n\log n (\log\log n)^3}{\epsilon^2}\right)$ comparisons for $\delta \geq \frac{1}{n}$, optimal up to a $(\log\log n)^3$ factor.
• Experiments show that the Knockout algorithm significantly outperforms variants of BTM-PAC and MallowsMPI, especially in low-$\epsilon$ regimes.
• The sample complexity of the Knockout algorithm increases almost linearly with $n$ and remains stable even as $\epsilon$ decreases, indicating robustness.
• The IR-BTM-PAC variant, with improved constants, shows performance gains but still underperforms the proposed Knockout algorithm.
• Theoretical analysis using Fano’s inequality confirms that the lower bound on sample complexity matches the upper bound up to logarithmic factors, validating the optimality of the proposed algorithms.