[Paper Review] QualiGPT: GPT as an easy-to-use tool for qualitative coding

He Zhang, Chuhao Wu+3 more|arXiv (Cornell University)|Oct 10, 2023

Computational and Text Analysis Methods18 citations

TL;DR

QualiGPT is an integrated GPT-based toolkit designed to simplify thematic analysis in qualitative research, improving usability, transparency, and accessibility compared to traditional CAQDAs and web ChatGPT.

ABSTRACT

Qualitative research delves deeply into individual complex perspectives on technology and various phenomena. However, a meticulous analysis of qualitative data often requires a significant amount of time, especially during the crucial coding stage. Although there is software specifically designed for qualitative evaluation, many of these platforms fall short in terms of automatic coding, intuitive usability, and cost-effectiveness. With the rise of Large Language Models (LLMs) such as GPT-3 and its successors, we are at the forefront of a transformative era for enhancing qualitative analysis. In this paper, we introduce QualiGPT, a specialized tool designed after considering challenges associated with ChatGPT and qualitative analysis. It harnesses the capabilities of the Generative Pretrained Transformer (GPT) and its API for thematic analysis of qualitative data. By comparing traditional manual coding with QualiGPT's analysis on both simulated and actual datasets, we verify that QualiGPT not only refines the qualitative analysis process but also elevates its transparency, credibility, and accessibility. Notably, compared to existing analytical platforms, QualiGPT stands out with its intuitive design, significantly reducing the learning curve and operational barriers for users.

Motivation & Objective

• Address the time and effort required for coding in qualitative analysis.
• Provide a user-friendly tool that leverages GPT for thematic analysis while mitigating common ChatGPT issues.
• Compare QualiGPT to traditional qualitative analysis software and to the web version of ChatGPT in terms usability, cost, and credibility.
• Enhance transparency, credibility, and accessibility of AI-assisted qualitative analysis.

Proposed method

• Develop QualiGPT as an integrated toolkit built on GPT API and tailored prompts for inductive thematic analysis.
• Incorporate prompts designed to reference original data to improve interpretability and traceability.
• Store prompts as presets to reduce prompt design workload and provide a visual, user-friendly interface.
• Evaluate QualiGPT on simulated and real datasets and compare its performance to manual coding.

Experimental results

Research questions

• RQ1 Can QualiGPT reduce the time and cost of qualitative data coding compared to manual methods?
• RQ2 Does QualiGPT improve transparency and credibility in AI-assisted qualitative analysis?
• RQ3 How does QualiGPT compare with traditional CAQDAs and the web ChatGPT in usability and learning curve?
• RQ4 What design practices (prompts, presets, data referencing) enhance the reliability of GPT-based thematic analysis?

Key findings

• QualiGPT streamlines the qualitative analysis workflow and reduces learning costs relative to traditional software.
• QualiGPT addresses transparency and credibility concerns by prompting for data references and interpretable outputs.
• QualiGPT offers improvements in usability, privacy protection, and performance over the web version of ChatGPT.
• QualiGPT provides a more intuitive user interface and lowers the coding time required for researchers compared to conventional software.