[论文解读] Incentives, Quality, and Risks: A Look Into the NSF Proposal Review Pilot

Parinaz Naghizadeh, Mingyan Liu|arXiv (Cornell University)|Jul 24, 2013

scientometrics and bibliometrics research参考文献 4被引用 11

一句话总结

本文評估了國家科學基金會（NSF）的試行性同行評審系統，其中申請人互相評審其提案，資金獲取機會與提案品質及評審品質掛鉤。該系統鼓勵誠實評審，但由於Borda計票機制與對共謀行為的防禦不足，可能導致非爭議性且平庸的提案獲得優勢。

ABSTRACT

The National Science Foundation (NSF) will be experimenting with a new distributed approach to reviewing proposals, whereby a group of principal investigators (PIs) or proposers in a subfield act as reviewers for the proposals submitted by the same set of PIs. To encourage honesty, PIs' chances for getting funded are tied to the quality of their reviews (with respect to the reviews provided by the entire group), in addition to the quality of their proposals. Intuitively, this approach can more fairly distribute the review workload, discourage frivolous proposal submission, and encourage high quality reviews. On the other hand, this method has already raised concerns about the integrity of the process and the possibility of strategic manipulation. In this paper, we take a closer look at three specific issues in an attempt to gain a better understanding of the strengths and limitations of the new process beyond first impressions and anecdotal evidence. We start by considering the benefits and drawbacks of bundling the quality of PIs' reviews with the scientific merit of their proposals. We then consider the issue of collusion and favoritism. Finally, we examine whether the new process puts controversial proposals at a disadvantage. We conclude that some benefits of using review quality as an incentive mechanism may outweigh its drawbacks. On the other hand, even a coalition of two PIs can cause significant harm to the process, as the built-in incentives are not strong enough to deter collusion. While we also confirm the common suspicion that the process is skewed toward non-controversial proposals, the more unexpected finding is that among equally controversial proposals, those of lower quality get a leg up through this process. Thus the process not only favors non-controversial proposals, but in some sense, mediocrity. We also discuss possible ways to improve this review process.

研究动机与目标

• 評估NSF試行的分散式同行評審系統，該系統將提案資助與科學價值及評審品質掛鉤。
• 調查將評審品質與提案評估結合是否能提升公平性與品質，同時最小化策略性操弄的風險。
• 評估該系統在共謀、偏袒及對爭議性或高風險提案的偏見方面的脆弱性。
• 識別激勵機制設計上的缺陷，這些缺陷可能破壞評審的完整性與公平性。
• 提出改進措施，如加強利益衝突管控、為爭議性提案設立專家小組，以及修改計分規則。

提出的方法

• 使用博弈論框架建模NSF評審試行計畫的激勵結構，分析誠實回報是否為穩定均衡。
• 採用基於Borda計票的排序方法，整合同行評審意見，並根據與集體評估的一致性計算審稿人獎勵積分。
• 模擬在各種情境下的評審結果，包括共謀情境與爭議性提案組合。
• 分析審稿人指派模式對獎勵積分分配與提案排序的影響。
• 提出修改建議，如事先指定利益衝突名單、限制相互評審，以及為高風險提案引入專家小組。
• 評估替代性激勵機制，包括將評審品質作為未來工作量或投稿資格的計價單位。

实验结果

研究问题

• RQ1 將提案資助與評審品質掛鉤是否能提供足夠激勵，促使審稿人進行誠實且高品質的評審？
• RQ2 兩位或以上PI之間的共謀在多大程度上會破壞評審過程的完整性？
• RQ3 Borda計票機制如何影響爭議性提案相較於非爭議性提案的排序？
• RQ4 為何在該系統下，低品質的爭議性提案會獲得相對優勢？
• RQ5 哪些設計變更可減輕對創新或高風險提案的偏見，同時不損害公平性？

主要发现

• 基於評審品質的激勵機制確實鼓勵誠實，但不具足夠韌性以阻止共謀，即使僅有兩位PI組成的共謀團體，也能顯著扭曲結果。
• 該系統系統性地不利於爭議性提案，因為審閱爭議性提案的審稿人因審稿人之間意見分歧而獲得較低獎勵積分。
• 在同等爭議性的提案中，低品質提案反而獲利，因其更可能獲得一小群審稿人的一致高分。
• 該系統無意間促進平庸化，因偏好避免爭議的提案，即使其科學價值僅屬中等。
• 審閱爭議性提案的審稿人不僅自身資金機會受損，其獎勵積分品質也因缺乏共識而受罰。
• 解決方案如為高風險提案設立專家小組、事先編列利益衝突名單，以及延長審稿期並匿名分享評論，有助於恢復公平性並檢測操弄行為。