[Paper Review] You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism

Weijie Su|arXiv (Cornell University)|Oct 27, 2021

Game Theory and Applications10 citations

TL;DR

This paper proposes the Isotonic Mechanism, a novel scoring approach that improves peer review accuracy by combining reviewers' noisy raw scores with the author's true ranking of their own papers. By solving a convex optimization problem that enforces isotonic (monotonic) constraints based on the author's ranking, the mechanism significantly enhances score accuracy and incentivizes truthful reporting under convex utility assumptions.

ABSTRACT

I consider a setting where reviewers offer very noisy scores for several items for the selection of high-quality ones (e.g., peer review of large conference proceedings), whereas the owner of these items knows the true underlying scores but prefers not to provide this information. To address this withholding of information, in this paper, I introduce the Isotonic Mechanism, a simple and efficient approach to improving imprecise raw scores by leveraging certain information that the owner is incentivized to provide. This mechanism takes the ranking of the items from best to worst provided by the owner as input, in addition to the raw scores provided by the reviewers. It reports the adjusted scores for the items by solving a convex optimization problem. Under certain conditions, I show that the owner's optimal strategy is to honestly report the true ranking of the items to her best knowledge in order to maximize the expected utility. Moreover, I prove that the adjusted scores provided by this owner-assisted mechanism are significantly more accurate than the raw scores provided by the reviewers. This paper concludes with several extensions of the Isotonic Mechanism and some refinements of the mechanism for practical consideration.

Motivation & Objective

• To address the growing problem of unreliable peer review scores in large machine learning conferences due to rising submission volumes and insufficient qualified reviewers.
• To design a mechanism that improves score accuracy without relying solely on reviewers, by leveraging information from paper authors.
• To create an incentive-compatible system where authors are motivated to truthfully report the ranking of their own papers.
• To develop a computationally efficient and theoretically grounded method that outperforms raw reviewer scores in estimating true paper quality.
• To extend the mechanism to practical challenges such as non-convex utilities, strategic behaviors, and multi-author papers.

Proposed method

• The Isotonic Mechanism uses a convex optimization framework to adjust raw reviewer scores based on the author-provided ranking of their own papers.
• It formulates the adjustment as an isotonic regression problem, ensuring that adjusted scores respect the order specified by the author’s ranking.
• The mechanism minimizes a loss function that balances fidelity to raw reviewer scores and adherence to the isotonic constraints from the author’s ranking.
• The optimization is parameterized by a penalty term λ, which controls the strength of the isotonic constraint; as λ→∞, the mechanism asymptotically recovers the isotonic solution.
• The method is designed so that under convex utility, the author’s optimal strategy is to truthfully report the ranking, ensuring incentive compatibility.
• The mechanism is orthogonal to existing approaches that focus on incentivizing reviewers, instead exploiting author-provided ordinal information.

Experimental results

Research questions

• RQ1 Can an author’s ranking of their own papers be used to significantly improve the accuracy of peer review scores?
• RQ2 Under what conditions is it optimal for an author to truthfully report the ranking of their own papers?
• RQ3 How does the Isotonic Mechanism compare to raw reviewer scores in terms of estimation accuracy, especially under high score variability?
• RQ4 Can the mechanism be extended to handle non-convex utility functions or strategic behaviors by authors?
• RQ5 How can the mechanism be adapted to multi-author papers where multiple authors may have conflicting incentives?

Key findings

• The Isotonic Mechanism produces adjusted scores that are significantly more accurate than raw reviewer scores, especially when reviewer score variability is high.
• Under convex utility, the author’s unique optimal strategy is to truthfully report the ranking of their papers, ensuring incentive compatibility.
• The mechanism asymptotically recovers the isotonic solution as the penalty parameter λ→∞, and this holds exactly for n=2 papers.
• The improvement in score accuracy is most substantial when the number of submissions per author is large and reviewer scores are highly variable—common in modern ML conferences.
• The mechanism remains effective even when authors are uncertain about the true ranking, as reporting the most accurate ranking available is still optimal.
• Empirical validation is encouraged, with NeurIPS 2021 having collected author rankings (though not used for decisions), providing a basis for future evaluation.