[Paper Review] Asymptotically Optimal Assignments In Ordinal Evaluations of Proposals

A. Yavuz Oruç, Abdullah Atmaca|ArXiv.org|Aug 22, 2009

Game Theory and Voting Systems17 references3 citations

TL;DR

This paper addresses the problem of optimally assigning proposals to referees in peer review systems to ensure every pair of proposals is compared at least once, under constraints on referee capacity. It establishes a general lower bound of n(n−1)/(k(k−1)) referees for k proposals per referee and presents an assignment scheme asymptotically matching this bound within a factor of 2, with exact solutions for specific k values like k = n/2, n/3, and n/4.

ABSTRACT

In ordinal evaluations of proposals in peer review systems, a set of proposals is assigned to a fixed set of referees so as to maximize the number of pairwise comparisons of proposals under certain referee capacity and proposal subject constraints. In this paper, the following two related problems are considered: (1) Assuming that each referee has a capacity to review k out of n proposals, 2 < k < n, determine the minimum number of referees needed to ensure that each pair of proposals is reviewed by at least one referee, (2) Find an assignment that meets the lower bound determined in (1). It is easy to see that one referee is both necessary and sufficient when k = n, and n(n-1)/2 referees are both necessary and sufficient when k = 2. We show that 6 referees are both necessary and sufficient when k = n/2. We further show that 11 referees are necessary and 12 are sufficient when k = n/3, and 18 referees are necessary and 20 referees are sufficient when k = n/4. A more general lower bound of n(n-1)/k(k-1) referees is also given for any k, 2 < k < n, and an assignment asymptotically matching this lower bound within a factor of 2 is presented. These results are not only theoretically interesting but they also provide practical methods for efficient assignments of proposals to referees.

Motivation & Objective

• To determine the minimum number of referees required so that every pair of n proposals is reviewed by at least one common referee, given each referee can review k proposals.
• To construct an assignment of proposals to referees that achieves this minimum number, ensuring full pairwise comparison coverage.
• To derive a general lower bound on the number of referees needed for any k satisfying 2 < k < n.
• To present an assignment strategy that asymptotically matches this lower bound within a factor of 2, with exact solutions for specific k values.
• To provide practical, efficient methods for real-world peer review systems with capacity and subject constraints.

Proposed method

• Derives a general lower bound of n(n−1)/(k(k−1)) referees using combinatorial arguments based on pairwise comparisons and referee capacity.
• Analyzes exact cases: 6 referees for k = n/2, 11 necessary and 12 sufficient for k = n/3, and 18 necessary and 20 sufficient for k = n/4.
• Constructs explicit assignment schemes that achieve the lower bound or come within a factor of 2 for general k.
• Uses combinatorial design principles to ensure that every pair of proposals is covered by at least one referee’s evaluation.
• Applies extremal combinatorics and graph-theoretic reasoning to model proposal pairs as edges and referees as hyperedges of size k.
• Validates the asymptotic optimality by showing that no assignment can use fewer than n(n−1)/(k(k−1)) referees, and the proposed scheme comes within a factor of 2 of this bound.

Experimental results

Research questions

• RQ1 What is the minimum number of referees required to ensure that every pair of n proposals is reviewed by at least one common referee, given each referee evaluates exactly k proposals?
• RQ2 Can an assignment be constructed that achieves this theoretical minimum for arbitrary k in the range 2 < k < n?
• RQ3 What are the exact minimum referee counts for specific values of k, such as k = n/2, k = n/3, and k = n/4?
• RQ4 How close can an assignment scheme come to the theoretical lower bound in the general case, and what is the asymptotic performance guarantee?
• RQ5 What practical assignment strategies can be derived that are both efficient and ensure full pairwise comparison coverage in peer review systems?

Key findings

• The general lower bound for the number of referees is n(n−1)/(k(k−1)), which is derived from the total number of pairwise comparisons and the maximum number of pairs each referee can cover.
• For k = n/2, exactly 6 referees are both necessary and sufficient to cover all pairs.
• For k = n/3, 11 referees are necessary and 12 are sufficient, showing a small gap between lower and upper bounds.
• For k = n/4, 18 referees are necessary and 20 are sufficient, again indicating tight bounds.
• An assignment scheme is presented that asymptotically matches the lower bound within a factor of 2 for any k satisfying 2 < k < n.
• The results provide a practical framework for designing efficient, scalable peer review systems with guaranteed coverage of all proposal pairs.