[Paper Review] Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing

Hongwei Li, Bin Yu|arXiv (Cornell University)|Nov 15, 2014

Mobile Crowdsensing and Crowdsourcing26 references61 citations

TL;DR

This paper proposes an iterative weighted majority voting (IWMV) method for crowdsourcing that optimizes finite-sample error rate bounds under the Dawid-Skene model. By iteratively refining worker weights to approximate the oracle MAP rule, IWMV achieves state-of-the-art accuracy with up to 100× lower computational cost than existing methods, demonstrating strong theoretical and empirical performance on real and simulated data.

ABSTRACT

Crowdsourcing has become an effective and popular tool for human-powered computation to label large datasets. Since the workers can be unreliable, it is common in crowdsourcing to assign multiple workers to one task, and to aggregate the labels in order to obtain results of high quality. In this paper, we provide finite-sample exponential bounds on the error rate (in probability and in expectation) of general aggregation rules under the Dawid-Skene crowdsourcing model. The bounds are derived for multi-class labeling, and can be used to analyze many aggregation methods, including majority voting, weighted majority voting and the oracle Maximum A Posteriori (MAP) rule. We show that the oracle MAP rule approximately optimizes our upper bound on the mean error rate of weighted majority voting in certain setting. We propose an iterative weighted majority voting (IWMV) method that optimizes the error rate bound and approximates the oracle MAP rule. Its one step version has a provable theoretical guarantee on the error rate. The IWMV method is intuitive and computationally simple. Experimental results on simulated and real data show that IWMV performs at least on par with the state-of-the-art methods, and it has a much lower computational cost (around one hundred times faster) than the state-of-the-art methods.

Motivation & Objective

• To derive finite-sample error rate bounds (in probability and expectation) for general aggregation rules in multi-class crowdsourcing under the Dawid-Skene model.
• To show that optimizing the mean error rate bound under the Homogenous Dawid-Skene model yields a rule close to the oracle MAP estimator.
• To develop an iterative, data-driven weighted majority voting method (IWMV) that approximates the oracle MAP rule with theoretical error rate guarantees.
• To demonstrate that IWMV achieves performance comparable to or better than state-of-the-art methods like EM-MAP and LPI, while being significantly faster.
• To provide a practical framework for computing error rate bounds for any decomposable aggregation rule using measurable quantities derived from the data.

Proposed method

• Derives finite-sample exponential bounds on error rates (in probability and expectation) for general aggregation rules under the Dawid-Skene model, applicable to multi-class labeling.
• Identifies that minimizing the mean error rate bound under the Homogenous Dawid-Skene model leads to a rule approximating the oracle MAP rule.
• Proposes an iterative weighted majority voting (IWMV) algorithm that updates worker weights in each iteration to minimize the error rate bound.
• Establishes a theoretical guarantee for the one-step version of IWMV, showing it improves upon majority voting in terms of error rate.
• Uses the EM algorithm as a baseline for comparison, while IWMV avoids complex iterative optimization by directly optimizing the error bound.
• Employs a two-stage approach: first compute key statistics (t₁, t₂, c, σ²) from data, then apply bounds and IWMV to estimate performance and labels.

Experimental results

Research questions

• RQ1 Can finite-sample error rate bounds be derived for general aggregation rules in multi-class crowdsourcing under the Dawid-Skene model?
• RQ2 Does minimizing the mean error rate bound under the Homogenous Dawid-Skene model yield a rule close to the oracle MAP estimator?
• RQ3 Can an iterative, data-driven weighted majority voting method be designed to approximate the oracle MAP rule with theoretical error rate guarantees?
• RQ4 How does the performance and computational cost of IWMV compare to state-of-the-art methods like EM-MAP and LPI in real-world and simulated settings?
• RQ5 To what extent do the theoretical error rate bounds reflect the actual error trends under varying worker reliability and task assignment configurations?

Key findings

• The error rate bound derived in the paper accurately reflects trends in the real error rate of the oracle MAP rule as key system parameters (e.g., number of workers, worker reliability) change.
• The one-step version of IWMV has a provable theoretical guarantee on error rate, outperforming standard majority voting.
• IWMV achieves performance on par with or better than the state-of-the-art EM-MAP and LPI methods on real datasets, including the Duchenne and Web search datasets.
• On the Duchenne dataset, IWMV outperformed EM-MAP and majority voting by at least 4% in error rate and ran ~100× faster than EM-MAP.
• IWMV is more robust to model misspecification than EM-based methods, maintaining stable performance under varying assumptions.
• The computational cost of IWMV is approximately 100 times lower than EM-MAP, making it highly efficient for large-scale crowdsourcing applications.