[Paper Review] Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment

Steven Jecmen, Nihar B. Shah+2 more|arXiv (Cornell University)|Jul 22, 2022

Expert finding and Q&A systems4 citations

TL;DR

This paper analyzes tradeoffs in preventing bid manipulation during reviewer assignment in academic conferences, proposing a framework of desiderata to evaluate existing and future algorithms. It evaluates approaches like Bid Limit, Random Display, Cycle Prevention, and Probability-Limited Randomized Assignment, showing that while each offers partial protection against manipulation, they involve tradeoffs between assignment quality, preference expressiveness, incentives to bid, and attack resilience.

ABSTRACT

Many conferences rely on paper bidding as a key component of their reviewer assignment procedure. These bids are then taken into account when assigning reviewers to help ensure that each reviewer is assigned to suitable papers. However, despite the benefits of using bids, reliance on paper bidding can allow malicious reviewers to manipulate the paper assignment for unethical purposes (e.g., getting assigned to a friend's paper). Several different approaches to preventing this manipulation have been proposed and deployed. In this paper, we enumerate certain desirable properties that algorithms for addressing bid manipulation should satisfy. We then offer a high-level analysis of various approaches along with directions for future investigation.

Motivation & Objective

• To identify and formalize key desiderata for reviewer assignment algorithms that balance bid-based assignment benefits with resistance to manipulation.
• To analyze existing mitigation strategies for bid manipulation in academic peer review, particularly in large-scale conferences.
• To evaluate tradeoffs between critical properties such as assignment quality, preference expressiveness, incentives to bid, and resistance to manipulation.
• To guide future algorithm design by highlighting gaps in current approaches and suggesting combinations of methods for improved robustness.
• To provide a structured framework for comparing and improving bid manipulation defenses in reviewer assignment systems.

Proposed method

• Proposes a set of five core desiderata: assignment quality, preference expressiveness, incentives to bid, low attack success rate, and high attack cost.
• Classifies existing mitigation techniques into categories: minimal-intrusion (e.g., Bid Limit, Random Display), similarity-based modeling (e.g., Bid Modeling, Reviewer Clustering), and randomized assignment (e.g., Probability-Limited Randomized Assignment).
• Analyzes how each method satisfies the desiderata, particularly focusing on how they limit manipulation while preserving reviewer preference expression.
• Introduces the Reviewer Clustering algorithm as a simplified variant of Bid Modeling to improve robustness against manipulation.
• Evaluates the impact of assumptions—such as the adversary’s ability to manipulate text similarities or subject areas—on algorithm effectiveness.
• Proposes hybrid approaches, such as combining Probability-Limited Randomized Assignment with Geographic Diversity, to enhance resilience.

Experimental results

Research questions

• RQ1 How do different mitigation strategies for bid manipulation in reviewer assignment trade off between assignment quality and resistance to manipulation?
• RQ2 To what extent do existing algorithms incentivize honest bidding while preventing malicious reviewers from increasing their assignment probability to target papers?
• RQ3 What are the relative strengths and weaknesses of minimal-intrusion, similarity-modeling, and randomized assignment approaches in defending against bid manipulation?
• RQ4 How can existing algorithms be combined to simultaneously improve multiple desiderata, such as attack resistance and assignment quality?
• RQ5 What assumptions about adversary capabilities significantly affect the effectiveness of bid manipulation defenses, and how can algorithms be designed to remain robust under worst-case assumptions?

Key findings

• Bid Limit and Random Display are minimally intrusive and preserve reviewer bidding behavior but offer limited protection against sophisticated manipulation.
• Bid Modeling and Reviewer Clustering improve attack resistance by leveraging non-bid similarity features, but their effectiveness degrades if adversaries can manipulate these features.
• Probability-Limited Randomized Assignment provides strong worst-case resilience by abstracting away similarity computation, but may lead reviewers to waste time bidding on ineligible papers.
• Random Display outperforms Probability-Limited Randomized Assignment in expected similarity due to pre-bidding randomization, ensuring reviewers only bid on eligible papers.
• Conferences like AAAI 2021 and 2022 have deployed multiple techniques simultaneously (e.g., Cycle Prevention, Geographic Diversity, Bid Limit), suggesting multi-method approaches are practical and effective.
• No single algorithm satisfies all desiderata; tradeoffs remain, and future work should focus on hybrid algorithms that combine strengths of existing methods.