[Paper Review] An Isotonic Mechanism for Overlapping Ownership

Jibang Wu, Haifeng Xu+2 more|arXiv (Cornell University)|Jun 19, 2023

Auction Theory and ApplicationsDecision Sciences31 references3 citations

TL;DR

This paper proposes an isotonic mechanism to elicit truthful self-evaluations from authors in overlapping ownership settings, such as multi-authored papers in scientific conferences. By partitioning papers into author-cohort blocks and applying isotonic regression to align rankings with raw review scores, the mechanism ensures truthfulness as a Nash equilibrium and achieves a constant-factor approximation via a greedy algorithm, significantly improving review quality in large-scale peer review systems.

ABSTRACT

Motivated by the problem of improving peer review at large scientific conferences, this paper studies how to elicit self-evaluations to improve review scores in a natural many-to-many owner-item (e.g., author-paper) situation with overlapping ownership. We design a simple, efficient and truthful mechanism to elicit self-evaluations from item owners that can be used to calibrate their noisy review scores in the existing evaluation process (e.g., papers' review scores from peers). Our approach starts by partitioning the owner-item relation structure into disjoint blocks, each sharing a common set of co-owners. We then elicit the ranking of items from each owner and employ isotonic regression to produce adjusted item scores, aligning with both the reported rankings and raw item review scores. We prove that truth-telling by all owners is a payoff dominant Nash equilibrium for any valid partition of the overlapping ownership sets under natural conditions. Moreover, the truthfulness depends on eliciting rankings independently within each block, making block partition optimization crucial for improving statistical efficiency. Despite being computationally intractable in general, we develop a nearly linear-time greedy algorithm that provably finds a performant block partition with appealing robust approximation guarantees. Extensive experiments on both synthetic data and real-world conference review data demonstrate the effectiveness of our mechanism in a pressing real-world problem.

Motivation & Objective

• To address the declining quality of peer review in large-scale machine learning conferences due to reviewer shortages and inconsistent scoring.
• To design a truthful mechanism that elicits reliable self-evaluations from authors of overlapping, co-owned papers.
• To ensure that truth-telling is a Nash equilibrium under the proposed mechanism, even when papers share multiple authors.
• To develop an efficient, nearly linear-time algorithm for partitioning authors into blocks that maximizes estimation efficiency.
• To evaluate the mechanism’s performance on both synthetic and real-world conference review data.

Proposed method

• Partition all conference submissions into disjoint blocks based on shared co-authorship, forming ownership sets.
• Elicit from each author a ranking of their own papers within each block, reflecting their subjective quality assessment.
• Apply isotonic regression to adjust raw review scores so they are consistent with the reported rankings while minimizing deviation in the least-squares sense.
• Prove that truth-telling is a Nash equilibrium under the mechanism when blocks are formed via valid partitioning of overlapping ownership sets.
• Design a greedy, nearly linear-time algorithm to approximate the optimal block partitioning that maximizes estimation efficiency.
• Establish theoretical approximation guarantees for the greedy algorithm, showing it achieves a constant-factor approximation ratio under polynomial objectives.

Figure 1: An example of an author-paper ownership set shown as a bipartite graph. An edge between an individual and a paper indicates that this individual is the author of the paper.

Experimental results

Research questions

• RQ1 Can a mechanism be designed to truthfully elicit self-evaluations from authors in overlapping ownership settings, such as multi-authored conference papers?
• RQ2 Under what conditions is truth-telling a Nash equilibrium in such a mechanism?
• RQ3 Is it necessary to partition authors into disjoint blocks and elicit only ranking information to ensure truthfulness?
• RQ4 Can an efficient algorithm approximate the optimal block partitioning with provable performance guarantees?
• RQ5 How does the proposed mechanism improve review score estimation accuracy compared to traditional peer review?

Key findings

• Truth-telling by all authors is a Nash equilibrium under the isotonic mechanism when papers are partitioned into disjoint ownership blocks based on co-authorship.
• The mechanism is provably truthful only if ranking information is elicited independently within each block, making block partitioning the sole optimization challenge.
• The proposed greedy algorithm achieves a constant-factor approximation of the optimal partition, with an approximation ratio of at most $1/α$ in the limit as the number of authors grows.
• The algorithm runs in nearly linear time, making it scalable for large conferences with tens of thousands of submissions.
• Experiments on synthetic and real-world conference data show that the owner-assisted calibration mechanism significantly improves estimation accuracy of ground-truth review scores.
• The mechanism outperforms baseline approaches by leveraging truthful self-evaluations from authors, especially in settings with high submission volume and reviewer scarcity.

Figure 2: The first two plots base on the ICLR 2022 dataset illustrate the probability of an paper getting accepted as an “oral”, “spotlight” “poster” w.r.t. its average review score. The dashed line denotes the estimated probability from raw data, the smooth line denotes the probability predicted b