[论文解读] Can Large Language Models emulate an inductive Thematic Analysis of semi-structured interviews? An exploration and provocation on the limits of the approach and the model

Stefano De Paoli|arXiv (Cornell University)|May 22, 2023

Computational and Text Analysis Methods被引用 12

一句话总结

本论文使用 GPT-3.5-Turbo 来模仿半结构化访谈数据的归纳性主题分析的某些方面，并讨论LLM能复制与否。

ABSTRACT

Large Language Models (LLMs) have emerged as powerful generative Artificial Intelligence solutions which can be applied to several fields and areas of work. This paper presents results and reflection of an experiment done to use the model GPT 3.5-Turbo to emulate some aspects of an inductive Thematic Analysis. Previous research on this subject has largely worked on conducting deductive analysis. Thematic Analysis is a qualitative method for analysis commonly used in social sciences and it is based on interpretations made by the human analyst(s) and the identification of explicit and latent meanings in qualitative data. Attempting an analysis based on human interpretation with an LLM clearly is a provocation but also a way to learn something about how these systems can or cannot be used in qualitative research. The paper presents the motivations for attempting this emulation, it reflects on how the six steps to a Thematic Analysis proposed by Braun and Clarke can at least partially be reproduced with the LLM and it also reflects on what are the outputs produced by the model. The paper used two existing datasets of open access semi-structured interviews, previously analysed with Thematic Analysis by other researchers. It used the previously produced analysis (and the related themes) to compare with the results produced by the LLM. The results show that the model can infer at least partially some of the main Themes. The objective of the paper is not to replace human analysts in qualitative analysis but to learn if some elements of LLM data manipulation can to an extent be of support for qualitative research.

研究动机与目标

• 动机并证明尝试使用 LLM 模拟归纳性主题分析的合理性。
• 研究 LLM 是否能够复现 Braun and Clarke 的六步主题分析的要素。
• 反思 LLM 在定性研究情境中的输出及局限性。

提出的方法

• 使用两份现有的开放获取的半结构化访谈数据集，之前由其他研究者用主题分析进行过分析。
• 将 GPT-3.5-Turbo 应用于模拟这些数据集的归纳性主题分析的某些方面。
• 将 LLM 获得的主题与先前的分析和主题进行比较，以评估部分对齐。
• 讨论模型输出的性质及其对定性研究实践的含义。

实验结果

研究问题

• RQ1 LLM 能否推断出先前半结构化访谈主题分析中识别的主要主题？
• RQ2 在多大程度上可以通过 LLM 重现或近似 Braun and Clarke 的六步主题分析？
• RQ3 使用 LLM 生成的输出来支持定性研究的优点与局限性是什么？

主要发现

• LLM 可以部分推断出数据集中的一些主要主题。
• 研究同时突显了 LLM 在定性研究任务中的潜在支持与显著局限。
• 目标不是取代人类分析师，而是探讨 LLM 的数据处理要素是否能够帮助定性研究。
• 分析促使人们反思 LLM 如何贡献于或使归纳性主题分析变得复杂。