[Paper Review] Algorithmic Ghost in the Research Shell: Large Language Models and Academic Knowledge Creation in Management Research

Nigel Williams, Stanislav Ivanov+1 more|arXiv (Cornell University)|Mar 10, 2023

Artificial Intelligence in Healthcare and Education8 citations

TL;DR

The paper reviews how large language models have been used to support academic knowledge creation beyond data analysis, based on a scoping review (2018–Jan 2023), and outlines future adoption pathways as Co-Writer, Research Assistant, and Respondent.

ABSTRACT

The paper looks at the role of large language models in academic knowledge creation based on a scoping review (2018 to January 2023) of how researchers have previously used the language model GPT to assist in the performance of academic knowledge creation tasks beyond data analysis. These tasks include writing, editing, reviewing, dataset creation and curation, which have been difficult to perform using earlier ML tools. Based on a synthesis of these papers, this study identifies pathways for a future academic research landscape that incorporates wider usage of large language models based on the current modes of adoption in published articles as a Co-Writer, Research Assistant and Respondent.

Motivation & Objective

• Assess how researchers have used large language models to assist in academic knowledge creation tasks beyond data analysis.
• Synthesize evidence from a scoping review (2018–January 2023) on GPT-assisted research activities.
• Identify and articulate prospective pathways for integrating LLMs into management research workflows.

Proposed method

• Conduct a scoping review of literature (2018–January 2023) on GPT-assisted academic tasks.
• Synthesize evidence across papers to map usage patterns of LLMs in knowledge creation tasks.
• Identify adoption modalities (Co-Writer, Research Assistant, Respondent) and discuss implications for future research.

Experimental results

Research questions

• RQ1 How have researchers used large language models to support academic knowledge creation tasks beyond data analysis?
• RQ2 What adoption modalities of LLMs emerge in published articles (Co-Writer, Research Assistant, Respondent)?
• RQ3 What future research landscape and pathways are suggested for integrating LLMs into management research?

Key findings

• LLMs have been used to assist tasks beyond data analysis such as writing, editing, reviewing, and dataset creation and curation.
• Published adoption patterns include three modes: Co-Writer, Research Assistant, and Respondent.
• The study identifies pathways for a future academic research landscape that incorporates wider usage of large language models in management research.