[论文解读] An Isotonic Mechanism for Overlapping Ownership

Jibang Wu, Haifeng Xu+2 more|arXiv (Cornell University)|Jun 19, 2023

Auction Theory and ApplicationsDecision Sciences参考文献 31被引用 3

一句话总结

本文提出了一种等序机制，用于在重叠所有权场景（如科学会议中的多作者论文）中获取作者的诚实自评。通过将论文划分为作者-同批区块，并对原始评审分数应用等序回归以使其与排名一致，该机制确保诚实作为纳什均衡成立，并通过贪心算法实现常数因子近似，显著提升了大规模同行评审系统中的评审质量。

ABSTRACT

Motivated by the problem of improving peer review at large scientific conferences, this paper studies how to elicit self-evaluations to improve review scores in a natural many-to-many owner-item (e.g., author-paper) situation with overlapping ownership. We design a simple, efficient and truthful mechanism to elicit self-evaluations from item owners that can be used to calibrate their noisy review scores in the existing evaluation process (e.g., papers' review scores from peers). Our approach starts by partitioning the owner-item relation structure into disjoint blocks, each sharing a common set of co-owners. We then elicit the ranking of items from each owner and employ isotonic regression to produce adjusted item scores, aligning with both the reported rankings and raw item review scores. We prove that truth-telling by all owners is a payoff dominant Nash equilibrium for any valid partition of the overlapping ownership sets under natural conditions. Moreover, the truthfulness depends on eliciting rankings independently within each block, making block partition optimization crucial for improving statistical efficiency. Despite being computationally intractable in general, we develop a nearly linear-time greedy algorithm that provably finds a performant block partition with appealing robust approximation guarantees. Extensive experiments on both synthetic data and real-world conference review data demonstrate the effectiveness of our mechanism in a pressing real-world problem.

研究动机与目标

• 为应对因评审人员短缺和评分不一致导致的大型机器学习会议同行评审质量下降问题。
• 设计一种诚实机制，以从具有重叠所有权的论文作者处获取可靠的自评。
• 确保在所提机制下，诚实陈述是纳什均衡，即使论文共享多位作者亦然。
• 开发一种高效、近乎线性时间的算法，将作者划分为区块，以最大化估计效率。
• 在合成数据和真实会议评审数据上评估该机制的性能。

提出的方法

• 根据共同作者关系将所有会议投稿划分为互不相交的区块，形成所有权集合。
• 从每位作者处获取其在每个区块内对自己论文的排名，反映其主观质量评估。
• 应用等序回归调整原始评审分数，使其与报告的排名一致，同时在最小二乘意义下最小化偏差。
• 证明当区块通过重叠所有权集合的有效划分形成时，诚实陈述是该机制下的纳什均衡。
• 设计一种贪心的、近乎线性时间的算法，以近似最优区块划分，从而最大化估计效率。
• 为该贪心算法建立理论近似保证，证明其在多项式目标下可实现常数因子近似比。

Figure 1: An example of an author-paper ownership set shown as a bipartite graph. An edge between an individual and a paper indicates that this individual is the author of the paper.

实验结果

研究问题

• RQ1 能否设计一种机制，在重叠所有权场景（如多作者会议论文）中诚实获取作者的自评？
• RQ2 在何种条件下，诚实陈述是该机制下的纳什均衡？
• RQ3 是否必须将作者划分为互不相交的区块，并仅获取排名信息，才能确保诚实性？
• RQ4 能否设计一种高效算法，以可证明的性能保证近似最优区块划分？
• RQ5 与传统同行评审相比，所提机制在多大程度上提升了评审分数估计的准确性？

主要发现

• 当论文根据共同作者关系被划分为互不相交的所有权区块时，所有作者的诚实陈述是等序机制下的纳什均衡。
• 仅当在每个区块内独立获取排名信息时，该机制才被证明是诚实的，因此区块划分成为唯一的优化挑战。
• 所提出的贪心算法可实现最优划分的常数因子近似，当作者数量趋于无穷时，近似比至多为 $1/α$。
• 该算法运行时间接近线性，使其可扩展至拥有数万篇投稿的大型会议。
• 在合成数据和真实会议数据上的实验表明，由作者协助的校准机制显著提升了对真实评审分数的估计准确性。
• 该机制通过利用作者提供的诚实自评，在高投稿量和评审人员稀缺的场景下，优于基线方法。

Figure 2: The first two plots base on the ICLR 2022 dataset illustrate the probability of an paper getting accepted as an “oral”, “spotlight” “poster” w.r.t. its average review score. The dashed line denotes the estimated probability from raw data, the smooth line denotes the probability predicted b