[Paper Review] Universal Clustering via Crowdsourcing

Ravi Kiran Raman, Lav R. Varshney|arXiv (Cornell University)|Oct 5, 2016

Mobile Crowdsensing and Crowdsourcing21 references3 citations

TL;DR

This paper proposes a universal clustering framework for crowdsourced data that operates without prior knowledge of worker reliability or task difficulty. By modeling worker responses through memory and distance traits, it designs asymptotically consistent clustering algorithms for both temporary and long-term workers, proving order-optimal sample complexity and establishing theoretical bounds on cost efficiency in unsupervised learning settings.

ABSTRACT

Consider unsupervised clustering of objects drawn from a discrete set, through the use of human intelligence available in crowdsourcing platforms. This paper defines and studies the problem of universal clustering using responses of crowd workers, without knowledge of worker reliability or task difficulty. We model stochastic worker response distributions by incorporating traits of memory for similar objects and traits of distance among differing objects. We are particularly interested in two limiting worker types---temporary workers who retain no memory of responses and long-term workers with memory. We first define clustering algorithms for these limiting cases and then integrate them into an algorithm for the unified worker model. We prove asymptotic consistency of the algorithms and establish sufficient conditions on the sample complexity of the algorithm. Converse arguments establish necessary conditions on sample complexity, proving that the defined algorithms are asymptotically order-optimal in cost.

Motivation & Objective

• To address the challenge of unsupervised clustering in crowdsourcing when worker reliability and task difficulty are unknown.
• To model human decision-making traits such as memory and perceptual distance among objects in worker responses.
• To design clustering algorithms that are universally applicable across different worker types—temporary and long-term—without prior channel knowledge.
• To establish asymptotic consistency and sample complexity bounds for clustering performance under universal settings.
• To prove that the proposed algorithms achieve order-optimal cost in terms of sample complexity, matching converse lower bounds.

Proposed method

• Modeling worker responses using a unified framework that incorporates memory effects and inter-object distance, with two limiting cases: temporary workers (i.i.d. responses) and long-term workers (Markov memory across tasks).
• Using a Markov memory model where the current response depends on the most recent response and the prior response to the same object class.
• Defining clustering algorithms based on minimum partition information functional and empirical entropy estimation to achieve universal clustering.
• Applying maximum likelihood estimation for entropy and mutual information to ensure asymptotic consistency under finite alphabet assumptions.
• Deriving convergence rates for empirical entropy and mutual information using concentration inequalities and total variation bounds.
• Establishing KL divergence bounds between hypotheses to prove constant separation between clusters, independent of sequence length.

Experimental results

Research questions

• RQ1 Can clustering be reliably performed in crowdsourcing without prior knowledge of worker reliability or task difficulty?
• RQ2 How do memory and perceptual distance among objects affect the design of universal clustering algorithms?
• RQ3 What is the minimum sample complexity required for asymptotically consistent clustering under universal settings?
• RQ4 Are the proposed algorithms order-optimal in terms of sample complexity, and can this be proven via converse arguments?
• RQ5 How do temporary workers (no memory) and long-term workers (with memory) differ in their clustering performance and cost efficiency?

Key findings

• The proposed clustering algorithms are asymptotically consistent, with empirical entropy and mutual information estimates converging to true values as sample size increases.
• The sample complexity for reliable clustering is bounded below by $ \frac{|\mathcal{X}|}{\epsilon \log |\mathcal{X}|} $, matching the known lower bound for entropy estimation.
• The algorithms achieve order-optimal sample complexity, as confirmed by converse arguments showing the lower bound matches the upper bound up to constant factors.
• For finite, constant alphabet sizes, the convergence rate of empirical entropy is bounded by $ \Pr[|\hat{H}(X)-H(X)|>\epsilon] \leq 2\exp\left(\frac{-n\epsilon^2}{2\log^2 n}+o(1)\right) $.
• The KL divergence between any two hypothesis distributions remains bounded by a constant independent of sequence length, ensuring stable clustering separation.
• The framework enables theoretical comparison between temporary and long-term worker pools, quantifying the trade-off between cost and reliability in crowdsourced clustering.