[论文解读] Unifying Framework for Crowd-sourcing via Graphon Estimation

Christina Lee, Devavrat Shah|arXiv (Cornell University)|Mar 23, 2017

Mobile Crowdsensing and Crowdsourcing被引用 3

一句话总结

本文提出了一种非参数的、基于图函数（graphon）的众包推理框架，通过将工作者-任务一致性概率建模为潜在特征上的潜在Lipschitz连续函数，统一了现有模型。通过将问题简化为图函数估计，该方法仅需每任务$ ilde{O}( ext{ln}(T)^{3/2})$次响应，即可在不了解底层函数或特征的情况下，以高概率正确推断出真实任务答案。

ABSTRACT

We consider the question of inferring true answers associated with tasks based on potentially noisy answers obtained through a micro-task crowd-sourcing platform such as Amazon Mechanical Turk. We propose a generic, non-parametric model for this setting: for a given task $i$, $1\leq i \leq T$, the response of worker $j$, $1\leq j\leq W$ for this task is correct with probability $F_{ij}$, where matrix $F = [F_{ij}]_{i\leq T, j\leq W}$ may satisfy one of a collection of regularity conditions including low rank, which can express the popular Dawid-Skene model; piecewise constant, which occurs when there is finitely many worker and task types; monotonic under permutation, when there is some ordering of worker skills and task difficulties; or Lipschitz with respect to an associated latent non-parametric function. This model, contains most, if not all, of the previously proposed models to the best of our knowledge. We show that the question of estimating the true answers to tasks can be reduced to solving the Graphon estimation problem, for which there has been much recent progress. By leveraging these techniques, we provide a crowdsourcing inference algorithm along with theoretical bounds on the fraction of incorrectly estimated tasks. Subsequently, we have a solution for inferring the true answers for tasks using noisy answers collected from crowd-sourcing platform under a significantly larger class of models. Concretely, we establish that if the $(i,j)$th element of $F$, $F_{ij}$, is equal to a Lipschitz continuous function over latent features associated with the task $i$ and worker $j$ for all $i, j$, then all task answers can be inferred correctly with high probability by soliciting $ ilde{O}(\ln(T)^{3/2})$ responses per task even without any knowledge of the Lipschitz function, task and worker features, or the matrix $F$.

研究动机与目标

• 将多种现有众包模型（如Dawid-Skene模型、低秩模型和分段常数模型）统一到一个非参数框架中。
• 解决从Amazon Mechanical Turk等平台收集的噪声大且可能存在偏差的响应中推断真实任务答案的挑战。
• 开发一种理论基础坚实的推理算法，实现在响应需求极少情况下的高概率正确性。

提出的方法

• 将工作者-任务一致性概率$F_{ij}$建模为与任务$i$和工作者$j$相关联的潜在特征的Lipschitz连续函数，从而实现非参数表示。
• 将真实答案估计问题简化为图函数估计问题，利用非参数图函数恢复的最新进展。
• 利用$F$矩阵在正则性条件（如低秩、分段常数、单调性或Lipschitz性）下的结构，统一并推广现有模型。
• 应用图函数估计技术，从未观测到的响应中恢复底层的一致性函数$F$。
• 推导在所提框架下错误估计任务比例的理论界。
• 在无需了解未知Lipschitz函数、潜在特征或矩阵$F$的情况下，实现每任务$ ilde{O}( ext{ln}(T)^{3/2})$的响应复杂度，以保证高概率正确推理。

实验结果

研究问题

• RQ1 是否可以设计一个单一的非参数框架，统一现有的众包推理模型，包括低秩、分段常数和单调模型？
• RQ2 如何将图函数估计技术适配以解决从噪声众包响应中推断真实答案的问题？
• RQ3 在不了解底层一致性函数或潜在特征的情况下，实现高概率正确推理所需的每任务最小响应数量是多少？
• RQ4 在工作者-任务一致性矩阵$F$满足何种正则性条件下，可以保证准确推理？

主要发现

• 所提框架在单一非参数形式下统一并推广了所有已知的众包参数模型，包括Dawid-Skene模型。
• 当一致性概率$F_{ij}$关于潜在特征为Lipschitz连续时，所有真实任务答案均可以高概率被正确推断。
• 即使在不了解Lipschitz函数、潜在特征或矩阵$F$的情况下，该方法也仅需$ ilde{O}( ext{ln}(T)^{3/2})$次响应每任务，即可实现高概率正确推理。
• 错误概率的理论界通过归约为图函数估计问题获得，而该问题近年来已取得显著进展。
• 该框架支持广泛的正则性条件，包括低秩、分段常数和单调结构，且在统一的估计范式下实现。
• 该方法在最小假设下实现鲁棒推理，使其适用于真实世界中工作者和任务特征未知的众包场景。