[Paper Review] Minimax Optimal Convergence Rates for Estimating Ground Truth from Crowdsourced Labels

Chao Gao, Dengyong Zhou|arXiv (Cornell University)|Oct 22, 2013

Mobile Crowdsensing and Crowdsourcing31 references61 citations

TL;DR

This paper establishes the first minimax optimal convergence rates for estimating ground truth labels from crowdsourced data using the Dawid-Skene estimator via a projected EM algorithm. It proves that the error rate decays exponentially fast, with the exponent determined by the collective wisdom of the crowd, and shows this rate is unimprovable in a minimax sense, resolving long-standing theoretical gaps in crowdsourcing estimation.

ABSTRACT

Crowdsourcing has become a primary means for label collection in many real-world machine learning applications. A classical method for inferring the true labels from the noisy labels provided by crowdsourcing workers is Dawid-Skene estimator. In this paper, we prove convergence rates of a projected EM algorithm for the Dawid-Skene estimator. The revealed exponent in the rate of convergence is shown to be optimal via a lower bound argument. Our work resolves the long standing issue of whether Dawid-Skene estimator has sound theoretical guarantees besides its good performance observed in practice. In addition, a comparative study with majority voting illustrates both advantages and pitfalls of the Dawid-Skene estimator.

Motivation & Objective

• To close the theoretical gap in understanding the statistical properties of the Dawid-Skene estimator, which has been widely used in practice but lacked formal analysis.
• To establish convergence rates for a projected EM algorithm used in estimating ground truth from noisy crowdsourced labels.
• To derive minimax lower bounds to prove the optimality of the convergence rate achieved by the estimator.
• To compare the Dawid-Skene estimator with majority voting, highlighting its advantages and limitations under model misspecification.
• To provide non-asymptotic bounds on estimation error for both label and worker ability estimation, and derive asymptotic distributions.

Proposed method

• Proposes a projected EM algorithm to iteratively estimate both ground truth labels and worker reliability parameters in the Dawid-Skene model.
• Uses a two-stage estimation process: E-step computes posterior probabilities of true labels given current worker reliability estimates, and M-step updates worker reliability parameters via maximum marginal likelihood.
• Applies non-asymptotic concentration inequalities and high-dimensional probability tools to derive bounds on estimation error in average and maximum losses.
• Derives a high-dimensional central limit theorem for the joint distribution of all worker ability estimates under finite-sample conditions.
• Employs lower bound arguments based on information-theoretic and minimax decision theory to prove the optimality of the convergence exponent.
• Constructs a concrete example with spammers to demonstrate the inconsistency of majority voting versus the robustness of the Dawid-Skene estimator under model assumptions.

Experimental results

Research questions

• RQ1 Is the Dawid-Skene estimator statistically consistent, and what is its convergence rate in estimating ground truth from crowdsourced labels?
• RQ2 Can the convergence rate of the projected EM algorithm for the Dawid-Skene estimator be characterized, and is it minimax optimal?
• RQ3 How does the Dawid-Skene estimator compare to majority voting in terms of consistency and robustness under model misspecification?
• RQ4 What are the non-asymptotic error bounds for estimating both ground truth labels and worker abilities?
• RQ5 What is the exact asymptotic distribution of the label estimator for any finite subset of workers, and what is the joint asymptotic distribution of all worker abilities?

Key findings

• The convergence rate of the Dawid-Skene estimator is exponentially small, with the exponent determined by the collective wisdom of the crowd, and this exponent is minimax optimal.
• The paper establishes a minimax lower bound that proves the convergence rate cannot be improved, confirming the theoretical optimality of the estimator.
• Non-asymptotic bounds show that the estimation error in average and maximum losses is bounded by $ O\left(\sqrt{\frac{\log m}{m}}\right) $, with high probability.
• The estimator achieves $ \|\hat{p}_i - p_i^*\| \leq O\left(\sqrt{\frac{\log m}{m}}\right) $ for worker ability estimates, uniformly over all workers.
• In a scenario with a majority of spammers, majority voting fails to converge, while the Dawid-Skene estimator still converges exponentially fast.
• The asymptotic distribution of the label estimator is derived for any finite subset of workers, and a high-dimensional central limit theorem is established for the joint distribution of all worker abilities.