[Paper Review] Do Large Language Models Reduce Research Novelty? Evidence from Information Systems Journals

Ali Safari|arXiv (Cornell University)|Mar 23, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

The paper measures semantic novelty of 13,847 Information Systems articles (2020–2025) using SPECTER2 embeddings to assess whether the ChatGPT release reduced novelty, finding a relative decline for authors from non-English-dominant countries.

ABSTRACT

Large language models such as ChatGPT have increased scholarly output, but whether this productivity boost produces genuine intellectual advancement remains untested. I address this gap by measuring the semantic novelty of 13,847 articles published between 2020 and 2025 in 44 Information Systems journals. Using SPECTER2 embeddings, I operationalize novelty as the cosine distance between each paper and its nearest prior neighbors. A difference-in-differences design with the November 2022 release of ChatGPT as the treatment break reveals a heterogeneous pattern: authors affiliated with institutions in non-English-dominant countries show a 0.18 standard deviation decline in relative novelty compared to authors in English-dominant countries (beta = -0.176, p < 0.001), equivalent to a 7-percentile-point drop in the novelty distribution. This finding is robust across alternative novelty specifications, treatment break dates, and sub-samples, and survives a placebo test at a pre-treatment break. I interpret these results through the lens of construal level theory, proposing that LLMs function as proximity tools that shift researchers from abstract, exploratory thinking toward concrete, convention-following execution. The paper contributes to the growing debate on whether LLM-driven productivity gains come at the cost of intellectual diversity.

Motivation & Objective

• Motivate the question of whether LLM-driven productivity impacts intellectual novelty in information systems scholarship.
• Quantify semantic novelty of IS articles across 44 journals from 2020–2025.
• Identify heterogeneity in novelty effects by author institutional language context.
• Test robustness of novelty effects to alternative specifications and placebo checks.

Proposed method

• Use SPECTER2 embeddings to compute novelty as cosine distance to nearest prior neighbors.
• Apply a difference-in-differences design with the November 2022 ChatGPT release as the treatment break.
• Analyze 13,847 IS articles published 2020–2025 to estimate causal impact on novelty by treatment timing.
• Examine robustness across alternative novelty measures, treatment dates, and sub-samples.

Experimental results

Research questions

• RQ1 Does the release of ChatGPT in November 2022 affect the semantic novelty of Information Systems research articles?
• RQ2 Is the novelty effect heterogeneous across authors from English-dominant vs non-English-dominant country institutions?
• RQ3 Is the observed effect robust to different novelty specifications and placebo tests?

Key findings

• Authors affiliated with institutions in non-English-dominant countries show a 0.18 standard deviation decline in relative novelty compared with English-dominant countries.
• The effect corresponds to a 7-percentage-point drop in the novelty distribution.
• The results are robust to alternative novelty specifications, treatment break dates, and sub-samples.
• A placebo test at a pre-treatment break does not generate the observed effect, supporting a causal interpretation.
• Interpreted through construal level theory as LLMs acting as proximity tools that shift thinking toward concrete execution.