[Paper Review] Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews

Weixin Liang, Zachary Izzo+10 more|arXiv (Cornell University)|Mar 11, 2024

Artificial Intelligence in Healthcare and Education63 citations

TL;DR

The paper introduces a scalable maximum likelihood method to estimate the fraction of text in a corpus substantially modified or generated by AI, and applies it to peer reviews from ML conferences after ChatGPT’s release, revealing a measurable rise in AI-modified reviews at ICLR, NeurIPS, and EMNLP but not in Nature journals.

ABSTRACT

We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place after the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text submitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated fraction of LLM-generated text is higher in reviews which report lower confidence, were submitted close to the deadline, and from reviewers who are less likely to respond to author rebuttals. We also observe corpus-level trends in generated text which may be too subtle to detect at the individual level, and discuss the implications of such trends on peer review. We call for future interdisciplinary work to examine how LLM use is changing our information and knowledge practices.

Motivation & Objective

• Motivate the need to measure AI-assisted text use at corpus level, not just per-document classifications.
• Develop a scalable statistical framework to estimate the AI-generated content fraction in large text corpora.
• Apply the framework to peer reviews from top ML conferences to observe post-ChatGPT trends.

Proposed method

• Frame the problem as estimating the mixture proportion alpha in (1-α)P + αQ where P is human-written text and Q is AI-generated text.
• Represent each document as a set of token occurrences (adjectives) to estimate P(t) and Q(t) via token presence frequencies.
• Use maximum likelihood estimation with log-likelihood L(α) = sum log((1-α)P(x_i) + αQ(x_i)).
• Generate training data by prompting human-written reviews and papers to create an AI-generated corpus for estimating P and Q.
• Estimate P and Q from token occurrence frequencies rather than full document likelihoods to improve tractability and robustness.
• Validate the method with semi-synthetic targets by sampling from human and AI validation corpora to verify α estimates.

Experimental results

Research questions

• RQ1 What fraction of peer review text in major ML conferences post-ChatGPT is substantially AI-generated or AI-modified?
• RQ2 How do corpus-level patterns of AI-modified text relate to reviewer behavior and content quality at scale?
• RQ3 Is the method robust to prompt variations and domain shifts across conferences and journals?
• RQ4 Can the approach distinguish substantial AI modification from simple proofreading?
• RQ5 What corpus-level signals (e.g., homogenization) accompany AI-generated content in reviews?

Key findings

• Estimated AI-modified content (α) rises post-ChatGPT for ML venues: ICLR 2024 reviews show about 10.6% and EMNLP 16.9% have substantial AI modification, while Nature reviews show no significant change.
• There is a deadline effect with higher AI usage in reviews submitted within 3 days of deadlines.
• Reviews with scholarly citations (e.g., containing “et al.”) show lower estimated AI usage than those without citations.
• Reviews with lower reply activity from authors tend to have higher estimated AI usage.
• Higher estimated AI modification correlates with homogenization in the embedding space of reviews, indicating corpus-level effects beyond individual documents.