[论文解读] What Can Natural Language Processing Do for Peer Review?

Ilia Kuznetsov, Osama Mohammed Afzal+22 more|TUbilio (Technical University of Darmstadt)|May 10, 2024

Topic Modeling被引用 5

一句话总结

该论文调查自然语言处理如何在AI会议评审的整个生命周期中协助同行评审，概述机会、挑战，以及行动号召。

ABSTRACT

The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In modern science, this process is largely delegated to peer review -- a distributed procedure in which each submission is evaluated by several independent experts in the field. Peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review -- manuscripts, reviews, discussions -- are largely text-based, Natural Language Processing has great potential to improve reviewing. As the emergence of large language models (LLMs) has enabled NLP assistance for many new tasks, the discussion on machine-assisted peer review is picking up the pace. Yet, where exactly is help needed, where can NLP help, and where should it stand aside? The goal of our paper is to provide a foundation for the future efforts in NLP for peer-reviewing assistance. We discuss peer review as a general process, exemplified by reviewing at AI conferences. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance, illustrated by existing work. We then turn to the big challenges in NLP for peer review as a whole, including data acquisition and licensing, operationalization and experimentation, and ethical issues. To help consolidate community efforts, we create a companion repository that aggregates key datasets pertaining to peer review. Finally, we issue a detailed call for action for the scientific community, NLP and AI researchers, policymakers, and funding bodies to help bring the research in NLP for peer review forward. We hope that our work will help set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.

研究动机与目标

• 绘制NLP在同行评审协助中的问题空间。
• 确定AI会议评审流程中NLP可以提供帮助的阶段。
• 突出NLP辅助同行评审在数据、伦理和评估方面的挑战。
• 提出一个伴随的数据存储库以及对社区行动的呼吁。

提出的方法

• 以AI会议同行评审流程作为持续示例的结构性演练。
• 讨论各阶段（预评审、评审中、评审后）潜在的NLP辅助任务。
• 结合现有工作进行说明，并就数据收集、许可和评估提供实用说明。
• 提出一个伴随仓库，汇集与NLP用于同行评审相关的数据集。
• 阐明指导未来工作的伦理、法律与方法论考量。

(a) Key stages of peer review

实验结果

研究问题

• RQ1 哪些NLP任务可以在同行评审过程的每个阶段提供有意义的支持？
• RQ2 在同行评审中部署NLP面临的主要挑战（数据、许可、评估、伦理）有哪些？
• RQ3 在不实现完全自动化的情况下，NLP工具如何提高同行评审的效率、质量和信任度？
• RQ4 推进机器辅助同行评审需要哪些基础设施、数据集和协作努力？
• RQ5 应当附带哪些最佳实践和政策来支持NLP支持的同行评审研究？

主要发现

• NLP可以在同行评审的多个阶段提供帮助，从提交筛选、评审者-论文匹配，到评审分析和元评审。
• 基于相似度的评审者-论文评分和基于关键词的匹配存在局限性和可解释性问题，需要进一步改进。
• 伦理、法律和数据挑战，包括偏见、透明度和许可，是在同行评审中部署NLP的核心问题。
• 提出一个伴随数据仓库，用于收集和分享与NLP用于同行评审相关的数据集，以促进社区合作。
• 全面自动化同行评审仍不太可能，但有针对性的NLP工具可以实质性减少工作量并提升流程鲁棒性。

Figure 2 : Areas of assistance before peer review.