[Paper Review] Bibliometric Analysis of Publisher and Journal Instructions to Authors on Generative-AI in Academic and Scientific Publishing

Conner Ganjavi, Michael Eppler+6 more|arXiv (Cornell University)|Jul 21, 2023

Machine Learning in Materials Science4 citations

TL;DR

This bibliometric study analyzes guidelines from the top 100 publishers and journals on generative AI (GAI) use in academic publishing, revealing widespread inconsistency in policies. Of the top 100 journals, 70% provided guidance, but only 17% of top publishers did, with significant variability in prohibitions, disclosure requirements, and access methods—highlighting a critical need for standardized, transparent GAI policies to preserve scientific integrity.

ABSTRACT

We aim to determine the extent and content of guidance for authors regarding the use of generative-AI (GAI), Generative Pretrained models (GPTs) and Large Language Models (LLMs) powered tools among the top 100 academic publishers and journals in science. The websites of these publishers and journals were screened from between 19th and 20th May 2023. Among the largest 100 publishers, 17% provided guidance on the use of GAI, of which 12 (70.6%) were among the top 25 publishers. Among the top 100 journals, 70% have provided guidance on GAI. Of those with guidance, 94.1% of publishers and 95.7% of journals prohibited the inclusion of GAI as an author. Four journals (5.7%) explicitly prohibit the use of GAI in the generation of a manuscript, while 3 (17.6%) publishers and 15 (21.4%) journals indicated their guidance exclusively applies to the writing process. When disclosing the use of GAI, 42.8% of publishers and 44.3% of journals included specific disclosure criteria. There was variability in guidance of where to disclose the use of GAI, including in the methods, acknowledgments, cover letter, or a new section. There was also variability in how to access GAI guidance and the linking of journal and publisher instructions to authors. There is a lack of guidance by some top publishers and journals on the use of GAI by authors. Among those publishers and journals that provide guidance, there is substantial heterogeneity in the allowable uses of GAI and in how it should be disclosed, with this heterogeneity persisting among affiliated publishers and journals in some instances. The lack of standardization burdens authors and threatens to limit the effectiveness of these regulations. There is a need for standardized guidelines in order to protect the integrity of scientific output as GAI continues to grow in popularity.

Motivation & Objective

• To assess the extent and content of generative-AI (GAI) guidance provided by top 100 academic publishers and journals.
• To identify patterns and disparities in GAI policies, including prohibitions, disclosure requirements, and guidance accessibility.
• To evaluate the standardization of GAI policies across affiliated publishers and journals.
• To highlight the risks of policy heterogeneity for authors and the integrity of scientific output.

Proposed method

• Systematic screening of publisher and journal websites from May 19–20, 2023, to identify GAI-related author instructions.
• Categorization of policies based on content: prohibitions, disclosure requirements, recommended disclosure locations, and guidance accessibility.
• Bibliometric analysis of policy prevalence and variation across the top 100 publishers and journals in science.
• Comparison of guidance across affiliated publisher-journal pairs to assess consistency.

Experimental results

Research questions

• RQ1 What proportion of the top 100 publishers and journals provide formal guidance on the use of generative AI by authors?
• RQ2 How do policies on generative AI use vary in terms of prohibitions, disclosure requirements, and recommended disclosure locations?
• RQ3 To what extent is there standardization in GAI policies across affiliated publishers and journals?
• RQ4 How accessible and consistently linked are GAI guidance documents across journal and publisher websites?

Key findings

• Among the top 100 publishers, only 17% provided guidance on generative-AI use, with 70.6% of those being in the top 25 publishers.
• Of the top 100 journals, 70% provided guidance, but only 42.8% of publishers and 44.3% of journals specified detailed disclosure criteria.
• 94.1% of publishers and 95.7% of journals prohibited the inclusion of generative AI as a co-author, while 5.7% of journals explicitly banned GAI use in manuscript generation.
• Disclosure locations varied widely, with mentions of methods, acknowledgments, cover letters, or new sections, and inconsistent linking of guidance between journals and publishers.
• Only 12 out of 17 publishers providing guidance were among the top 25, indicating limited policy development among major publishing houses.
• Substantial heterogeneity in GAI policies persists even among affiliated publishers and journals, undermining clarity and enforceability.