[Paper Review] Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model

Stefano De Paoli|arXiv (Cornell University)|May 22, 2023

Computational and Text Analysis Methods12 citations

TL;DR

The paper experiments with GPT-3.5-Turbo to emulate aspects of inductive thematic analysis on semi-structured interview data and discusses what can and cannot be replicated by an LLM.

ABSTRACT

Large Language Models (LLMs) have emerged as powerful generative Artificial Intelligence solutions which can be applied to several fields and areas of work. This paper presents results and reflection of an experiment done to use the model GPT 3.5-Turbo to emulate some aspects of an inductive Thematic Analysis. Previous research on this subject has largely worked on conducting deductive analysis. Thematic Analysis is a qualitative method for analysis commonly used in social sciences and it is based on interpretations made by the human analyst(s) and the identification of explicit and latent meanings in qualitative data. Attempting an analysis based on human interpretation with an LLM clearly is a provocation but also a way to learn something about how these systems can or cannot be used in qualitative research. The paper presents the motivations for attempting this emulation, it reflects on how the six steps to a Thematic Analysis proposed by Braun and Clarke can at least partially be reproduced with the LLM and it also reflects on what are the outputs produced by the model. The paper used two existing datasets of open access semi-structured interviews, previously analysed with Thematic Analysis by other researchers. It used the previously produced analysis (and the related themes) to compare with the results produced by the LLM. The results show that the model can infer at least partially some of the main Themes. The objective of the paper is not to replace human analysts in qualitative analysis but to learn if some elements of LLM data manipulation can to an extent be of support for qualitative research.

Motivation & Objective

• Motivate and justify attempting to emulate inductive thematic analysis with an LLM.
• Investigate whether an LLM can reproduce elements of Braun and Clarke's six-step thematic analysis.
• Reflect on the outputs and limitations of the LLM in qualitative research contexts.

Proposed method

• Use two existing open-access semi-structured interview datasets previously analyzed with thematic analysis by other researchers.
• Apply GPT-3.5-Turbo to emulate aspects of the inductive thematic analysis on these datasets.
• Compare the LLM-derived themes with the previously produced analysis and themes to assess partial alignment.
• Discuss the nature of outputs produced by the model and what they imply for qualitative research practice.

Experimental results

Research questions

• RQ1 Can an LLM infer the main themes identified in prior thematic analyses of semi-structured interviews?
• RQ2 To what extent can the six steps of Braun and Clarke's thematic analysis be reproduced or approximated by an LLM?
• RQ3 What are the strengths and limitations of using LLM-generated outputs to support qualitative research?

Key findings

• The LLM can partially infer some of the main themes from the datasets.
• The study highlights both potential support and notable limits of LLMs for qualitative research tasks.
• The objective is not to replace human analysts but to explore whether LLM data manipulation elements can aid qualitative research.
• The analysis prompts reflection on how LLMs can contribute to or complicate inductive thematic analysis.