[Paper Review] Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations

Lixiang Yan, Vanessa Echeverría+6 more|arXiv (Cornell University)|Nov 7, 2023

Computational and Text Analysis Methods4 citations

TL;DR

This study investigates how qualitative researchers collaborate with ChatGPT in thematic analysis, finding it enhances coding efficiency and data comprehension despite concerns over trust and accuracy. It proposes five actionable design recommendations to improve transparency, contextual understanding, feedback loops, and validation in human-AI collaboration for qualitative research.

ABSTRACT

Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT. Through a user study involving ten qualitative researchers, we found ChatGPT to be a valuable collaborator for thematic analysis, enhancing coding efficiency, aiding initial data exploration, offering granular quantitative insights, and assisting comprehension for non-native speakers and non-experts. Yet, concerns about its trustworthiness and accuracy, reliability and consistency, limited contextual understanding, and broader acceptance within the research community persist. We contribute five actionable design recommendations to foster effective human-AI collaboration. These include incorporating transparent explanatory mechanisms, enhancing interface and integration capabilities, prioritising contextual understanding and customisation, embedding human-AI feedback loops and iterative functionality, and strengthening trust through validation mechanisms.

Motivation & Objective

• To understand qualitative researchers' experiences collaborating with ChatGPT in thematic analysis.
• To identify benefits and challenges in using generative AI for qualitative data coding and theme development.
• To derive actionable design recommendations for improving human-AI collaboration in qualitative research workflows.
• To address concerns around trust, reliability, and contextual understanding in GenAI-assisted thematic analysis.

Proposed method

• Conducted a user study with ten experienced qualitative researchers using ChatGPT for thematic analysis on a 30-utterance interview transcript.
• Collected qualitative data through semi-structured interviews focusing on user perceptions, workflow integration, and challenges.
• Analyzed feedback to identify recurring themes related to efficiency, comprehension, trust, and interface limitations.
• Synthesized five design recommendations based on user insights and observed collaboration dynamics.
• Focused on iterative, user-driven feedback loops and contextual customization to enhance AI responsiveness.
• Emphasized validation mechanisms and transparent explanations to improve user trust in AI-generated outputs.

Experimental results

Research questions

• RQ1 How do qualitative researchers perceive their collaboration with ChatGPT during thematic analysis?
• RQ2 What benefits and challenges do researchers experience when using ChatGPT for coding and theme generation?
• RQ3 How can human-AI collaboration in thematic analysis be designed to enhance efficiency, accuracy, and user trust?
• RQ4 What interface and interaction features are most valuable for integrating GenAI into qualitative research workflows?

Key findings

• ChatGPT significantly improved coding efficiency and supported initial data exploration, particularly for non-native speakers and non-experts.
• Researchers valued ChatGPT’s ability to generate granular, data-driven insights and assist in theme refinement through iterative interaction.
• Persistent concerns were raised about the reliability and consistency of AI outputs, especially regarding factual accuracy and contextual understanding.
• Users expressed a strong need for transparent explanatory mechanisms, such as coverage metrics and side-by-side comparisons with human coding.
• The current ChatGPT interface was perceived as suboptimal for thematic analysis, with limited integration into existing research workflows.
• Feedback loops and contextual customization were identified as critical for enabling iterative, user-guided AI collaboration.