[Paper Review] An Analysis of Rank Aggregation Algorithms

Gattaca Lv|arXiv (Cornell University)|Feb 21, 2014

Game Theory and Voting Systems13 references3 citations

TL;DR

This paper proposes a fixed-parameter tractable dynamic programming algorithm for computing exact Kemeny consensus rankings, demonstrating that NP-hard rank aggregation becomes efficient when rankings are similar (low average pairwise Kendall-Tau distance). The method efficiently computes optimal Kemeny rankings on real-world university data, outperforming heuristic and Borda methods when similarity is high.

ABSTRACT

Rank aggregation is an essential approach for aggregating the preferences of multiple agents. One rule of particular interest is the Kemeny rule, which maximises the number of pairwise agreements between the final ranking and the existing rankings. However, Kemeny rankings are NP-hard to compute. This has resulted in the development of various algorithms. Fortunately, NP-hardness may not reflect the difficulty of solving problems that arise in practice. As a result, we aim to demonstrate that the Kemeny consensus can be computed efficiently when aggregating different rankings in real case. In this paper, we extend a dynamic programming algorithm originally for Kemeny scores. We also provide details on the implementation of the algorithm. Finally, we present results obtained from an empirical comparison of our algorithm and two other popular algorithms based on real world and randomly generated problem instances. Experimental results show the usefulness and efficiency of the algorithm in practical settings.

Motivation & Objective

• To address the computational intractability of Kemeny consensus in rank aggregation by exploring practical tractability.
• To evaluate whether NP-hardness of Kemeny ranking computation is a barrier in real-world settings with similar rankings.
• To implement and empirically compare a fixed-parameter dynamic programming algorithm with Borda count and a heuristic method.
• To determine under what conditions exact Kemeny consensus can be computed efficiently in practice.
• To assess the impact of ranking similarity (measured by average pairwise Kendall-Tau distance) on algorithmic performance.

Proposed method

• Extends a fixed-parameter dynamic programming algorithm originally designed for Kemeny scores to compute exact Kemeny consensus rankings.
• Uses the average pairwise Kendall-Tau distance as a key parameter, exploiting its influence on computational complexity.
• Implements a 3D dynamic programming table where state transitions consider candidate assignments to ranks based on pairwise agreement.
• Employs pruning techniques to limit the number of candidate universities per rank, bounded by 4d, where d is the average pairwise distance.
• Employs empirical evaluation on real university rankings and randomly generated data with varying levels of similarity (d).
• Compares performance against Borda count and a heuristic algorithm in terms of running time and solution quality.

Experimental results

Research questions

• RQ1 Can exact Kemeny consensus be computed efficiently in practice when rankings are sufficiently similar?
• RQ2 How does the average pairwise Kendall-Tau distance affect the runtime of the dynamic programming algorithm?
• RQ3 How does the proposed dynamic programming algorithm compare in performance and accuracy to Borda count and heuristic methods?
• RQ4 Under what conditions does the dynamic programming approach become infeasible due to memory constraints?
• RQ5 To what extent can the algorithm scale to larger datasets while maintaining exactness?

Key findings

• The dynamic programming algorithm computes exact Kemeny consensus efficiently when the average pairwise Kendall-Tau distance is low, such as in real-world university rankings.
• The algorithm's performance is significantly better on datasets with high similarity (e.g., SAME_m) than on random or dissimilar ones (e.g., DIFF_m, RANDOM_m).
• For datasets with low d (d ≤ m/2), the algorithm remains efficient regardless of the number of candidates m, indicating fixed-parameter tractability with respect to d.
• The running time of the dynamic programming algorithm increases exponentially with m when d is large, as seen in RANDOM_m where m increases from 6 to 12.
• The algorithm becomes infeasible when d ≥ m due to excessive memory consumption, as the required table size grows to O(2^m · m²), exceeding 2GB for m ≥ 21.
• The Borda count and heuristic methods remain efficient even for larger m, but only the dynamic programming approach guarantees exact Kemeny optimality when d is small.