[Paper Review] Auctions and Prediction Markets for Scientific Peer Review

Siddarth Srinivasan, Jamie Morgenstern|arXiv (Cornell University)|Aug 27, 2021

Auction Theory and Applications4 citations

TL;DR

This paper proposes a two-stage mechanism combining VCG auctions and a prediction market-style system (H-DIPP) to improve scientific peer review by incentivizing high-quality submissions and reviews. Authors bid for review slots, and reviewers are rewarded based on review quality, using auction revenues to fund incentives, thereby addressing reviewer shortages and low effort through mechanism design.

ABSTRACT

Peer reviewed publications are considered the gold standard in certifying and disseminating ideas that a research community considers valuable. However, we identify two major drawbacks of the current system: (1) the overwhelming demand for reviewers due to a large volume of submissions, and (2) the lack of incentives for reviewers to participate and expend the necessary effort to provide high-quality reviews. In this work, we adopt a mechanism-design approach to propose improvements to the peer review process. We present a two-stage mechanism which ties together the paper submission and review process, simultaneously incentivizing high-quality reviews and high-quality submissions. In the first stage, authors participate in a VCG auction for review slots by submitting their papers along with a bid that represents their expected value for having their paper reviewed. For the second stage, we propose a novel prediction market-style mechanism (H-DIPP) building on recent work in the information elicitation literature, which incentivizes participating reviewers to provide honest and effortful reviews. The revenue raised by the Stage I auction is used in Stage II to pay reviewers based on the quality of their reviews.

Motivation & Objective

• To address the growing demand for peer reviewers due to high submission volumes in scientific publishing.
• To solve the lack of incentives for reviewers to provide high-quality, effortful reviews.
• To design a mechanism that aligns incentives for both authors and reviewers through economic mechanisms.
• To integrate submission and review processes into a unified, incentive-compatible system.
• To use revenue from a VCG auction to fund high-quality reviews via a prediction market-style mechanism.

Proposed method

• Authors submit papers with a bid representing their expected value for review, participating in a VCG auction to secure review slots.
• The VCG auction determines which papers are selected for review based on bids and social welfare maximization.
• A novel mechanism called H-DIPP (Honest-Decision Prediction Market) is used in the second stage to elicit honest and effortful reviews.
• Reviewers report their confidence and assessment of papers in a prediction market format, with payouts based on prediction accuracy.
• Auction revenues from Stage I are used to fund reviewer payments in Stage II, creating a self-sustaining incentive system.
• The mechanism ensures truthfulness and effort through proper scoring rules and incentive alignment in the H-DIPP framework.

Experimental results

Research questions

• RQ1 How can we incentivize authors to submit high-quality papers in a system with limited review capacity?
• RQ2 How can we ensure reviewers provide honest and effortful evaluations without external rewards?
• RQ3 Can a prediction market mechanism effectively elicit high-quality reviews in a scientific peer review context?
• RQ4 How can auction mechanisms for review slots be designed to maximize social welfare while funding reviewer incentives?
• RQ5 What is the impact of revenue recycling from auctions on the quality and honesty of peer reviews?

Key findings

• The proposed two-stage mechanism successfully aligns incentives for both authors and reviewers through economic mechanisms.
• The VCG auction efficiently allocates review slots based on authors' bids, reflecting perceived value of review.
• The H-DIPP mechanism incentivizes honest and effortful reviews by linking reviewer payouts to prediction accuracy.
• Revenue from the auction is effectively recycled to fund reviewer compensation, creating a self-sustaining system.
• The mechanism design ensures truthfulness and effort in reviews without relying on external enforcement or reputation systems.
• The integration of auctions and prediction markets offers a scalable and incentive-compatible alternative to traditional peer review.