[Paper Review] Is ChatGPT Transforming Academics' Writing Style?

Mingmeng Geng, Roberto Trotta|arXiv (Cornell University)|Apr 12, 2024

Artificial Intelligence in Healthcare and Education5 citations

TL;DR

The paper analyzes a million arXiv abstracts to detect ChatGPT-imprinted writing style through word-frequency changes, estimating ChatGPT impact by category and time, with CS showing the strongest effect (~35% under a simple prompt).

ABSTRACT

Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT's writing style in their abstracts through a statistical analysis of word frequency changes. Our model is calibrated and validated on a mixture of real abstracts and ChatGPT-modified abstracts (simulated data) after a careful noise analysis. The words used for estimation are not fixed but adaptive, including those with decreasing frequency. We find that large language models (LLMs), represented by ChatGPT, are having an increasing impact on arXiv abstracts, especially in the field of computer science, where the fraction of LLM-style abstracts is estimated to be approximately 35%, if we take the responses of GPT-3.5 to one simple prompt, "revise the following sentences", as a baseline. We conclude with an analysis of both positive and negative aspects of the penetration of LLMs into academics' writing style.

Motivation & Objective

• Motivate and quantify whether ChatGPT influences academic writing styles in arXiv abstracts.
• Develop a statistical framework to detect ChatGPT-like word-frequency fingerprints over time.
• Calibrate and validate the method using real and ChatGPT-modified (simulated) abstracts.
• Estimate the density of ChatGPT-influenced text across disciplines and time.
• Discuss implications, benefits, and risks of ChatGPT’s penetration into scholarly writing.

Proposed method

• Define a change factor R_i to measure word-frequency shifts over time (Equation 1).
• Use ChatGPT-driven simulations by polishing real abstracts with simple prompts to estimate word-change rates r̂_ij (Equation 2).
• Model ChatGPT impact with an η_j(t) term representing the share of abstracts affected (Equation 5).
• Incorporate noise via δ_ij and construct a bias-aware loss L_j,t(η_j) to estimate η_j (Equations 18–23).
• Calibrate word sets I_j and test robustness with varying prompts and mixing ratios (Equations 35–37).
• Calibrate f*_ij(t) using pre-ChatGPT periods and validate with GPT-3.5–driven simulations (Section 4).

Figure 1: The 12 words with the highest change rate $R_{i}$ and satisfying $\max_{t}(f_{i}(t))>500$ . The vertical red dashed line demarcates the first time period after ChatGPT’s release.

Experimental results

Research questions

• RQ1 Can a statistical signature in word frequencies reveal ChatGPT’s impact on arXiv abstracts?
• RQ2 How does ChatGPT influence word usage across disciplines and over time?
• RQ3 What is the estimated density of ChatGPT-style writing in different fields, especially CS?
• RQ4 How robust is the estimation to different prompts and calibration choices?
• RQ5 What are the limitations and potential biases in measuring ChatGPT influence via word frequencies?

Key findings

• ChatGPT-style text penetration is detectable in arXiv abstracts after its release, with computer science showing the strongest uptake.
• Estimated ChatGPT impact in CS is about 35% using a simple prompt baseline (“revise the following sentences”).
• Word-frequency shifts reflect both topic trends (e.g., COVID-19, LLMs, AI) and non-topic changes (e.g., function words like “are”/“is”).
• Words like “significant” show substantial increases in simulated ChatGPT processing across several categories (CS, math, astro, cond-mat).
• A calibration-based, transparent frequency-analysis approach can quantify ChatGPT’s impact without relying on black-box detectors.

Figure 2: Examples of words with rapidly growing frequency in arXiv abstracts.