[论文解读] Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis

He Zhang, Chuhao Wu+4 more|arXiv (Cornell University)|Sep 19, 2023

Computational and Text Analysis Methods被引用 62

一句话总结

本文研究提示设计是否能够提升 ChatGPT 在定性主题分析中的表现，并开发提示框架以提高效率与质量。它结合先导研究、实验和定性方法来评估接受度与实际指南。

ABSTRACT

AI tools, particularly large-scale language model (LLM) based applications such as ChatGPT, have the potential to simplify qualitative research. Through semi-structured interviews with seventeen participants, we identified challenges and concerns in integrating ChatGPT into the qualitative analysis process. Collaborating with thirteen qualitative researchers, we developed a framework for designing prompts to enhance the effectiveness of ChatGPT in thematic analysis. Our findings indicate that improving transparency, providing guidance on prompts, and strengthening users' understanding of LLMs' capabilities significantly enhance the users' ability to interact with ChatGPT. We also discovered and revealed the reasons behind researchers' shift in attitude towards ChatGPT from negative to positive. This research not only highlights the importance of well-designed prompts in LLM applications but also offers reflections for qualitative researchers on the perception of AI's role. Finally, we emphasize the potential ethical risks and the impact of constructing AI ethical expectations by researchers, particularly those who are novices, on future research and AI development.

研究动机与目标

• Address the labor-intensive nature of thematic analysis in qualitative research.
• Examine the potential of ChatGPT to augment thematic analysis tasks.
• Develop cueing frameworks to improve ChatGPT's performance in qualitative coding.
• Assess researchers' acceptance, challenges, and strategies for using AI in qualitative analysis.

提出的方法

• Conducted a reflexive thematic analysis of interviews and experiments across three components: pilot study, prompt-design experiment, and post-study questionnaires.
• Used a corpus of focus-group data (remote work) generated by ChatGPT to test prompts and coding performance.
• Collaborated with qualitative analysts to craft cueing frameworks guiding ChatGPT-assisted analysis.
• Compared experiences with qualitative analysis software (e.g., NVivo, atlas.ti) and AI-assisted approaches.
• Applied Technology Acceptance Model (TAM) in post-study questionnaires to gauge usefulness and ease of use.
• Iteratively designed prompts with participants to identify strategies that enhance accuracy, transparency, and efficiency.

实验结果

研究问题

• RQ1 Can the performance of ChatGPT in qualitative analysis tasks be enhanced through prompt design? If so, how?

主要发现

• Cueing frameworks co-developed with qualitative analysts improved ChatGPT’s contribution to thematic analysis and connected AI more effectively with qualitative methods.
• Participants reported benefits of ChatGPT for rapid data processing, overview generation, and preliminary insights, despite concerns.
• Main concerns identified include transparency, accuracy, prompt design difficulty, and reviewing prompt-generated results.
• Pilot and experimental phases highlighted a trade-off between automation benefits and the need for interpretability and verifiability.
• The study emphasizes the importance of prompt-design frameworks over single-purpose prompts for robust AI-assisted qualitative analysis.