[论文解读] Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing

Ashish Khetan, Sewoong Oh|arXiv (Cornell University)|Feb 10, 2016

Mobile Crowdsensing and Crowdsourcing参考文献 28被引用 29

一句话总结

本文提出了一种自适应任务分配方案，通过在广义Dawid-Skene模型下实时推断任务难度和工作者可靠性，动态分配标签，实现了预算最优。证明了自适应方案显著优于非自适应方案，实现相同准确率所需预算可减少至$\lambda / \lambda_{\text{min}}$倍，并提出了一种匹配的高效算法，达到极小化误差率的理论极限。

ABSTRACT

Crowdsourcing platforms provide marketplaces where task requesters can pay to get labels on their data. Such markets have emerged recently as popular venues for collecting annotations that are crucial in training machine learning models in various applications. However, as jobs are tedious and payments are low, errors are common in such crowdsourced labels. A common strategy to overcome such noise in the answers is to add redundancy by getting multiple answers for each task and aggregating them using some methods such as majority voting. For such a system, there is a fundamental question of interest: how can we maximize the accuracy given a fixed budget on how many responses we can collect on the crowdsourcing system. We characterize this fundamental trade-off between the budget (how many answers the requester can collect in total) and the accuracy in the estimated labels. In particular, we ask whether adaptive task assignment schemes lead to a more efficient trade-off between the accuracy and the budget. Adaptive schemes, where tasks are assigned adaptively based on the data collected thus far, are widely used in practical crowdsourcing systems to efficiently use a given fixed budget. However, existing theoretical analyses of crowdsourcing systems suggest that the gain of adaptive task assignments is minimal. To bridge this gap, we investigate this question under a strictly more general probabilistic model, which has been recently introduced to model practical crowdsourced annotations. Under this generalized Dawid-Skene model, we characterize the fundamental trade-off between budget and accuracy. We introduce a novel adaptive scheme that matches this fundamental limit. We further quantify the fundamental gap between adaptive and non-adaptive schemes, by comparing the trade-off with the one for non-adaptive schemes. Our analyses confirm that the gap is significant.

研究动机与目标

• 为弥合实际自适应众包系统与理论分析之间的差距，后者表明自适应带来的收益有限。
• 刻画在众包二值分类中预算（收集标签的数量）与准确率之间的基本权衡。
• 设计一种高效的自适应任务分配方案，在广义Dawid-Skene模型下匹配理论上的极小化误差率。
• 量化自适应与非自适应方案在预算效率方面的根本性能差距。
• 使用比以往工作更一般化的概率模型，为自适应方案提供严格的理论基础。

提出的方法

• 本文使用包含异质任务难度和工作者可靠性的广义Dawid-Skene模型对众包标注过程进行建模。
• 提出一种自适应任务分配策略，根据对潜在参数的实时推断，为估计难度更高的任务分配更多工作者。
• 设计一种高效推理算法，通过收集到的响应迭代估计工作者可靠性和任务难度，从而实现动态重分配。
• 该方案使用线性规划松弛来指导任务分配，同时保持对误差率的理论保证。
• 理论分析推导出自适应与非自适应方案的极小化误差率，揭示了根本性的预算差距。
• 分析基于中心极限定理的渐近近似，结合次高斯消息分布的尾部界。

实验结果

研究问题

• RQ1 自适应任务分配方案能否在众包中显著优于非自适应方案，实现更好的预算-准确率权衡？
• RQ2 在广义Dawid-Skene模型下，给定预算的准确率基本极限是什么？
• RQ3 自适应与非自适应方案之间的性能差距如何随任务难度和工作者可靠性变化？
• RQ4 能否设计一种高效的自适应方案，使其匹配理论上的极小化误差率？
• RQ5 实时参数估计在提升预算效率方面起到什么作用？

主要发现

• 所提出的自适应方案达到了极小化误差率，证明其在基本预算-准确率权衡下是最优的。
• 自适应与非自适应方案之间的根本差距由因子$\lambda / \lambda_{\text{min}}$量化，其中$\lambda_{\text{min}}$严格小于$\lambda$。
• 非自适应方案为达到与自适应方案相同的准确率，所需预算高出$\lambda / \lambda_{\text{min}}$倍。
• 自适应方案的充分预算条件为$\Gamma \geq C \frac{m}{\lambda_{\text{min}} \lambda \sigma^2} (\log(1/\varepsilon))^{3/2}$，相比先前工作更优，优势达$\sqrt{\log(1/\varepsilon)}$倍。
• 分析结果证实，自适应显著提升了预算效率，尤其在任务难度差异较大时更为明显。
• 实验结果表明，算法3中的参数估计算法在实践中表现良好，但其理论样本复杂度尚未得到证明。