[论文解读] A Truth Serum for Large-Scale Evaluations

Vijay Kamble, David Marn+3 more|arXiv (Cornell University)|Jul 25, 2015

Mobile Crowdsensing and Crowdsourcing参考文献 18被引用 8

一句话总结

本文提出了一种用于大规模评估的新型奖励机制，通过仅在答案稀少时奖励与同伴的一致性来激励诚实回答——将更高的奖励分配给较少见的回答。该机制证明了诚实报告是严格贝叶斯-纳什均衡，并在期望收益上近似最优，且在评估规模增大时，任何更优的均衡都会变得完全信息充分。

ABSTRACT

A major challenge in obtaining large-scale evaluations, e.g., product or service reviews on online platforms, labeling images, grading in online courses, etc., is that of eliciting honest responses from agents in the absence of verifiability. We propose a new reward mechanism with strong incentive properties applicable in a wide variety of such settings. This mechanism has a simple and intuitive output agreement structure: an agent gets a reward only if her response for an evaluation matches that of her peer. But instead of the reward being the same across different answers, it is inversely proportional to a popularity index of each answer. This index is a second order population statistic that captures how frequently two agents performing the same evaluation agree on the particular answer. Rare agreements thus earn a higher reward than agreements that are relatively more common. In the regime where there are a large number of evaluation tasks, we show that truthful behavior is a strict Bayes-Nash equilibrium of the game induced by the mechanism. Further, we show that the truthful equilibrium is approximately optimal in terms of expected payoffs to the agents across all symmetric equilibria, where the approximation error vanishes in the number of evaluation tasks. Moreover, under a mild condition on strategy space, we show that any symmetric equilibrium that gives a higher expected payoff than the truthful equilibrium must be close to being fully informative if the number of evaluations is large. These last two results are driven by a new notion of an agreement measure that is shown to be monotonic in information loss. This notion and its properties are of independent interest.

研究动机与目标

• 解决在无法验证真实性的大规模评估中获取诚实回答的挑战。
• 设计一种在多样化评估场景中均保持强激励特性的奖励机制。
• 确保在大规模设置下，诚实行为是严格贝叶斯-纳什均衡。
• 表明诚实均衡在所有对称均衡中近似最优。
• 建立在温和条件下，优于诚实均衡的均衡必须近乎完全信息充分。

提出的方法

• 该机制仅在代理的回答与同伴一致时才给予奖励，但奖励与答案的流行度指数成反比。
• 流行度指数是一种二阶统计量，用于衡量在评估中两个代理对某一答案达成一致的频率。
• 稀有答案——即一致频率较低的答案——会获得更高的奖励，从而创造诚实报告的激励。
• 该机制通过博弈论工具在大规模制度下进行分析，重点研究贝叶斯-纳什均衡。
• 引入了一种新的共识度量，其在信息损失上单调，从而可分析均衡效率。
• 理论分析依赖于评估任务数量趋于无穷时的渐近性质。

实验结果

研究问题

• RQ1 能否设计一种奖励机制，在无法验证真实性的大规模评估中激励诚实报告？
• RQ2 在该机制下，诚实报告是否为严格贝叶斯-纳什均衡？
• RQ3 诚实均衡的期望收益与其它对称均衡相比如何？
• RQ4 高于诚实的均衡必须具备何种结构特性？
• RQ5 新共识度量与评估系统中的信息损失有何关系？

主要发现

• 在大规模制度下，诚实报告是所提机制中的严格贝叶斯-纳什均衡。
• 在所有对称均衡中，诚实均衡的期望收益近似最优，且近似误差随评估数量增加而消失。
• 任何对称均衡若其期望收益高于诚实均衡，则在评估数量较大时必须接近完全信息充分。
• 所提出的共识度量在信息损失上单调，为分析评估系统中的信息效率提供了新工具。
• 该机制的激励结构对同伴回答的选择具有鲁棒性，且无需验证真实标签。
• 该机制通过基于答案相对稀有性的奖励结构，实现了强激励相容性。