[Paper Review] Building reputation systems for better ranking

Luo-Luo Jiang, Matúš Medo+3 more|arXiv (Cornell University)|Jan 13, 2010

Peer-to-Peer Network Technologies5 references3 citations

TL;DR

This paper proposes an iterative reputation-based ranking algorithm that jointly estimates user reputation and object quality in rating systems using weighted averaging of ratings, where weights are derived from users' consistency with estimated object qualities. The method significantly improves ranking accuracy—evidenced by higher AUC scores—on both synthetic and real-world data (MovieLens and Amazon), outperforming simple average rating across varying rating densities and benchmark object sets.

ABSTRACT

How to rank web pages, scientists and online resources has recently attracted increasing attention from both physicists and computer scientists. In this paper, we study the ranking problem of rating systems where users vote objects by discrete ratings. We propose an algorithm that can simultaneously evaluate the user reputation and object quality in an iterative refinement way. According to both the artificially generated data and the real data from MovieLens and Amazon, our algorithm can considerably enhance the ranking accuracy. This work highlights the significance of reputation systems in the Internet era and points out a way to evaluate and compare the performances of different reputation systems.

Motivation & Objective

• To address the limitation of simple average rating in rating systems, where unreliable or inconsistent users can distort object rankings.
• To develop a joint estimation method for user reputation and object quality that iteratively refines both metrics using rating consistency.
• To evaluate the algorithm’s performance on real-world data using known high-quality benchmark objects (e.g., Oscar and National Book Award winners).
• To provide a quantitative benchmark for comparing reputation-based ranking algorithms using AUC as a performance metric.
• To demonstrate that reputation-aware ranking outperforms naive averaging, especially in sparse rating environments.

Proposed method

• The algorithm estimates object quality $ q_k $ as a weighted average of user ratings, where weights are proportional to $ \xi_i^{-\alpha} $, with $ \xi_i $ being the mean square deviation of user $ i $'s ratings from estimated object qualities.
• User reputation is quantified via $ \xi_i = \frac{1}{M_i} \sum_k (x_{ik} - q_k)^2 $, representing the average error of user $ i $'s ratings against estimated object qualities.
• The system iteratively updates $ q_k $ and $ \xi_i $ in alternating steps, starting with $ \xi_i = 1 $, until convergence is reached.
• The parameter $ \alpha \geq 0 $ controls the sensitivity of weight assignment to reputation, with $ \alpha = 0 $ reducing to the simple average method.
• The method is tested on artificial data with uniform distributions and real datasets (MovieLens and Amazon), using AUC to measure performance relative to known benchmark objects.
• For real data, benchmark objects are defined as Oscar-winning movies (74) and National Book Award-winning books (189), with AUC computed as the probability that a benchmark object ranks higher than a non-benchmark one.

Experimental results

Research questions

• RQ1 Can a joint iterative estimation of user reputation and object quality improve ranking accuracy compared to simple average rating?
• RQ2 How does the performance of the reputation-based algorithm vary with different values of the weighting parameter $ \alpha $?
• RQ3 Does the algorithm maintain superior performance across varying rating densities in real-world datasets?
• RQ4 To what extent does the algorithm outperform baseline methods in identifying known high-quality objects (e.g., award winners) in MovieLens and Amazon?
• RQ5 Can AUC serve as a reliable and meaningful metric for comparing reputation-based ranking algorithms on real data?

Key findings

• The proposed algorithm significantly improves ranking accuracy, as measured by AUC, compared to the simple average rating method (i.e., $ \alpha = 0 $) on both MovieLens and Amazon datasets.
• On MovieLens, the AUC value increases from approximately 0.5 (random) to over 0.65 with optimal $ \alpha $, indicating a strong improvement in identifying Oscar-winning films.
• On Amazon, the AUC value also exceeds 0.6 for optimal $ \alpha $, demonstrating consistent performance gains in identifying National Book Award-winning books.
• The algorithm performs better on denser rating networks, but the qualitative improvement over average rating remains consistent across different rating densities.
• The optimal $ \alpha $ value varies between datasets, suggesting that the algorithm's performance is sensitive to data structure and distribution.
• The iterative refinement process converges reliably, and the method effectively identifies and downweights inconsistent raters, enhancing object quality estimation.