[Paper Review] Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

Cheng Tan, Dongxin Lyu+6 more|arXiv (Cornell University)|Jun 9, 2024

Interdisciplinary Research and CollaborationDecision Sciences3 citations

TL;DR

This paper reformulates academic peer review as a multi-turn, long-context dialogue with distinct roles for authors, reviewers, and decision makers, introducing the ReviewMT dataset of 26,841 papers with 92,017 annotated reviews. It demonstrates that supervised fine-tuning on this dataset significantly improves LLM performance in generating realistic, role-specific reviews and decisions, especially in multi-turn interactions.

ABSTRACT

Large Language Models (LLMs) have demonstrated wide-ranging applications across various fields and have shown significant potential in the academic peer-review process. However, existing applications are primarily limited to static review generation based on submitted papers, which fail to capture the dynamic and iterative nature of real-world peer reviews. In this paper, we reformulate the peer-review process as a multi-turn, long-context dialogue, incorporating distinct roles for authors, reviewers, and decision makers. We construct a comprehensive dataset containing over 26,841 papers with 92,017 reviews collected from multiple sources, including the top-tier conference and prestigious journal. This dataset is meticulously designed to facilitate the applications of LLMs for multi-turn dialogues, effectively simulating the complete peer-review process. Furthermore, we propose a series of metrics to evaluate the performance of LLMs for each role under this reformulated peer-review setting, ensuring fair and comprehensive evaluations. We believe this work provides a promising perspective on enhancing the LLM-driven peer-review process by incorporating dynamic, role-based interactions. It aligns closely with the iterative and interactive nature of real-world academic peer review, offering a robust foundation for future research and development in this area. We open-source the dataset at https://github.com/chengtan9907/ReviewMT.

Motivation & Objective

• To address the static, one-shot nature of existing LLM applications in peer review by modeling the process as a dynamic, iterative dialogue.
• To capture the full complexity of real-world peer review, including rebuttals, feedback loops, and role-specific responsibilities.
• To construct a large-scale, multi-source dataset (ReviewMT) with annotated multi-turn dialogues from top venues like ICLR and Nature Communications.
• To develop role-specific evaluation metrics that fairly assess LLM performance across reviewers, authors, and decision makers.
• To demonstrate that supervised fine-tuning on ReviewMT significantly enhances LLMs' ability to generate accurate, context-aware, and role-appropriate peer review responses.

Proposed method

• Reformulate peer review as a multi-turn, long-context dialogue involving three distinct roles: authors, reviewers, and decision makers.
• Construct the ReviewMT dataset by collecting real peer review interactions from ICLR (ICLR) and Nature Communications (NC), including full review histories and rebuttals.
• Annotate each dialogue turn with role-specific context, ensuring full manuscript grounding and iterative interaction tracking.
• Design a suite of evaluation metrics tailored to each role: review hit rate, decision hit rate, F1-score, and text similarity for response quality.
• Apply supervised fine-tuning on the ReviewMT dataset to improve LLM performance in generating role-specific responses.
• Evaluate LLMs using both zero-shot and fine-tuned settings on two test splits: ReviewMT-ICLR (multi-turn) and ReviewMT-NC (one-turn), with quantitative comparison across models.

Experimental results

Research questions

• RQ1 Can peer review be effectively modeled as a multi-turn, role-based dialogue that captures the iterative and interactive nature of real academic review?
• RQ2 How does the inclusion of multi-turn interactions and long-context manuscript grounding affect LLM performance in generating realistic peer review responses?
• RQ3 To what extent can supervised fine-tuning on the ReviewMT dataset improve LLMs’ ability to generate accurate, contextually appropriate, and role-specific reviews and decisions?
• RQ4 How do different LLM architectures perform across the three roles (reviewer, author, decision maker) in the context of multi-turn peer review dialogues?
• RQ5 What is the impact of dataset structure—particularly multi-turn vs. one-turn dialogue—on the performance gains from fine-tuning?

Key findings

• Supervised fine-tuning on the ReviewMT-ICLR dataset significantly improves LLM performance, with the best model achieving a 94.20% F1-score on the reviewer role and 81.11% decision hit rate.
• The Gemma model achieves the highest decision hit rate (81.11%) and strong F1-score (0.9420), indicating strong performance in role-specific decision-making.
• Zero-shot performance is consistently low, with decision hit rates below 25% across most models, highlighting the necessity of fine-tuning for realistic peer review generation.
• The ReviewMT-ICLR dataset, with its multi-turn structure, enables greater performance gains from fine-tuning compared to the one-turn ReviewMT-NC dataset, underscoring the value of dialogue complexity.
• Text similarity metrics show that fine-tuned models produce responses that are significantly closer to human-generated reviews in style and content, especially in multi-turn settings.
• The Yuan model achieves a perfect 1.0000 F1-score but only 0.05% decision hit rate, suggesting it may be overly constrained in decision-making despite strong review quality.