[Paper Review] Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction

Jacob Steinhardt, Gregory Valiant+1 more|arXiv (Cornell University)|Jun 16, 2016

Mobile Crowdsensing and Crowdsourcing36 references3 citations

TL;DR

This paper proposes a robust matrix completion framework for identifying top-rated items in adversarial crowdsourcing settings, where only a small fraction of raters are reliable. Using semidefinite programming with nuclear norm constraints and incorporating a few personal ratings to break symmetry, the method recovers the top β-fraction of items with error at most ϵ, even under adversarial perturbations, while requiring work independent of the total number of items n.

ABSTRACT

We consider a crowdsourcing model in which $n$ workers are asked to rate the quality of $n$ items previously generated by other workers. An unknown set of $αn$ workers generate reliable ratings, while the remaining workers may behave arbitrarily and possibly adversarially. The manager of the experiment can also manually evaluate the quality of a small number of items, and wishes to curate together almost all of the high-quality items with at most an $ε$ fraction of low-quality items. Perhaps surprisingly, we show that this is possible with an amount of work required of the manager, and each worker, that does not scale with $n$: the dataset can be curated with $ ilde{O}\Big(\frac{1}{βα^3ε^4}\Big)$ ratings per worker, and $ ilde{O}\Big(\frac{1}{βε^2}\Big)$ ratings by the manager, where $β$ is the fraction of high-quality items. Our results extend to the more general setting of peer prediction, including peer grading in online classrooms.

Motivation & Objective

• To address the challenge of identifying high-quality items in crowdsourcing systems where a subset of raters may be adversarial or unreliable.
• To enable accurate identification of the top β-fraction of items using only a small number of personal ratings from the user, breaking symmetry between honest and dishonest raters.
• To design a scalable algorithm that requires work per rater independent of the total number of items n, depending only on reliability fraction α and desired accuracy ϵ.
• To relax assumptions on inter-rater agreement, allowing for varying rater strictness as long as expected ratings induce similar rankings.

Proposed method

• Formulates the recovery problem as a semidefinite program maximizing the inner product ⟨~A, M⟩ subject to constraints on matrix entries, row sums, and nuclear norm.
• Imposes a nuclear norm bound ∥M∥∗ ≤ 2αϵαβnm to promote low-rank structure and suppress noise from unreliable raters.
• Uses a small set of personal ratings ~r to break symmetry between honest and adversarial raters, which is essential when α < 1/2.
• Employs matrix completion techniques adapted from stochastic block models and community detection, tailored to the sparse, constant-degree setting.
• Relies on a weaker form of inter-rater agreement—ranking consistency rather than exact rating agreement—enabling robustness to rater bias.

Experimental results

Research questions

• RQ1 Can we reliably identify the top β-fraction of items in a crowdsourced rating system when a significant fraction of raters are adversarial?
• RQ2 How can we break symmetry between honest and adversarial raters without relying on ground-truth labels?
• RQ3 What is the minimal amount of work required per rater to achieve ϵ-approximate recovery of the top items, independent of n?
• RQ4 To what extent can the method tolerate rater bias or varying strictness while still recovering the correct ranking?

Key findings

• With high probability (at least 99%), the algorithm recovers βn items whose average quality is at most ϵ worse than the true top set T∗.
• The number of ratings k per rater required is O(1/βα³ϵ⁴), which grows slowly with desired accuracy and reliability.
• The number of items k₀ that the user must rate personally is Ω(log(1/αβϵ)/(βϵ²)), which is information-theoretically necessary and nearly optimal.
• The method remains effective even when reliable raters differ in strictness, as long as their expected ratings induce approximately the same ranking.
• The nuclear norm constraint plays a critical role in suppressing adversarial influence and enabling recovery under weak inter-rater agreement.