[Paper Review] Crowdsourcing via Pairwise Co-occurrences: Identifiability and Algorithms

Shahana Ibrahim, Xiao Fu+2 more|arXiv (Cornell University)|Sep 26, 2019

Sparse and Compressive Sensing Techniques39 references21 citations

TL;DR

This paper proposes a novel framework for crowdsourcing label aggregation using pairwise co-occurrences of annotator responses, enabling provable identifiability of the Dawid-Skene model with lower sample complexity than tensor-based methods. By leveraging second-order statistics and algebraic algorithms, the approach achieves superior performance in identifying confusion matrices and ground-truth priors, even under sparse labeling and unreliable annotators, outperforming state-of-the-art baselines.

ABSTRACT

The data deluge comes with high demands for data labeling. Crowdsourcing (or, more generally, ensemble learning) techniques aim to produce accurate labels via integrating noisy, non-expert labeling from annotators. The classic Dawid-Skene estimator and its accompanying expectation maximization (EM) algorithm have been widely used, but the theoretical properties are not fully understood. Tensor methods were proposed to guarantee identification of the Dawid-Skene model, but the sample complexity is a hurdle for applying such approaches---since the tensor methods hinge on the availability of third-order statistics that are hard to reliably estimate given limited data. In this paper, we propose a framework using pairwise co-occurrences of the annotator responses, which naturally admits lower sample complexity. We show that the approach can identify the Dawid-Skene model under realistic conditions. We propose an algebraic algorithm reminiscent of convex geometry-based structured matrix factorization to solve the model identification problem efficiently, and an identifiability-enhanced algorithm for handling more challenging and critical scenarios. Experiments show that the proposed algorithms outperform the state-of-art algorithms under a variety of scenarios.

Motivation & Objective

• To address the high sample complexity of tensor-based methods in identifying the Dawid-Skene model for crowdsourced labeling.
• To develop a framework using pairwise co-occurrences (second-order statistics) instead of third-order statistics to improve sample efficiency.
• To ensure model identifiability under realistic conditions, such as the presence of a well-trained annotator or partial labeling.
• To design efficient and robust algorithms for large-scale crowdsourcing applications with provable convergence and accuracy.
• To outperform existing state-of-the-art methods in label aggregation accuracy under diverse and challenging labeling scenarios.

Proposed method

• Utilizes pairwise co-occurrence statistics between annotators’ responses to estimate joint response distributions, avoiding reliance on third-order statistics.
• Employs an algebraic algorithm inspired by convex geometry and structured matrix factorization to efficiently solve the model identification problem.
• Introduces an identifiability-enhanced iterative optimization algorithm for handling cases with no highly reliable annotators.
• Relies on the assumption that at least one annotator has a sufficiently high reliability (e.g., confusion matrix close to identity) to enable identifiability.
• Uses concentration inequalities to bound the sample complexity for reliable estimation of pairwise co-occurrences under random sampling of data by annotators.
• Derives theoretical guarantees on identifiability by analyzing the geometric structure of response distributions in the probability simplex.

Experimental results

Research questions

• RQ1 Can the Dawid-Skene model be identified using only pairwise co-occurrences of annotator responses, without relying on third-order statistics?
• RQ2 What conditions ensure the identifiability of the ground-truth label prior and annotator confusion matrices under the proposed pairwise co-occurrence framework?
• RQ3 How does the sample complexity of the proposed method compare to tensor-based approaches in practice, especially under sparse labeling?
• RQ4 Can the proposed algebraic algorithm achieve high accuracy and scalability in large-scale crowdsourcing settings?
• RQ5 How does the identifiability-enhanced iterative algorithm improve performance in scenarios with no highly reliable annotators?

Key findings

• The proposed framework achieves model identifiability under realistic conditions, such as the existence of a well-trained annotator, by using only second-order statistics.
• The sample complexity for estimating pairwise co-occurrences is significantly lower than for third-order statistics, especially when annotator labeling probability is low.
• The algebraic algorithm provides efficient, scalable solutions suitable for large-scale crowdsourcing with provable convergence.
• The identifiability-enhanced iterative algorithm improves robustness in challenging scenarios, such as when no highly reliable annotators exist.
• Experiments demonstrate that both proposed algorithms outperform state-of-the-art methods across various labeling scenarios and data sparsity levels.
• Theoretical bounds show that with sufficient annotators (e.g., at least 1 + (2^{K-4}(K-1)^2)/(K α_min^{2(K-2)} ε^2) log(K(K-1)/ρ)), the model is ε-sufficiently scattered and identifiable with high probability.