[Paper Review] Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysis

Han Meng, Yitian Yang+4 more|arXiv (Cornell University)|May 9, 2024

Mental Health via Writing4 citations

TL;DR

This paper introduces CHALET, a human-LLM collaborative framework that enhances theory-driven qualitative analysis by leveraging LLMs for data collection, deductive coding, and collaborative inductive coding on disagreement cases. Applied to mental illness stigma, CHALET uncovered implicit stigmatization themes across cognitive, emotional, and behavioral dimensions, demonstrating LLMs' potential to generate novel conceptual insights beyond mere coding accuracy.

ABSTRACT

Qualitative coding is a demanding yet crucial research method in the field of Human-Computer Interaction (HCI). While recent studies have shown the capability of large language models (LLMs) to perform qualitative coding within theoretical frameworks, their potential for collaborative human-LLM discovery and generation of new insights beyond initial theory remains underexplored. To bridge this gap, we proposed CHALET, a novel approach that harnesses the power of human-LLM partnership to advance theory-driven qualitative analysis by facilitating iterative coding, disagreement analysis, and conceptualization of qualitative data. We demonstrated CHALET's utility by applying it to the qualitative analysis of conversations related to mental-illness stigma, using the attribution model as the theoretical framework. Results highlighted the unique contribution of human-LLM collaboration in uncovering latent themes of stigma across the cognitive, emotional, and behavioral dimensions. We discuss the methodological implications of the human-LLM collaborative approach to theory-based qualitative analysis for the HCI community and beyond.

Motivation & Objective

• To address the underexplored potential of LLMs in advancing qualitative analysis beyond basic coding accuracy.
• To develop a methodological framework that leverages human-LLM collaboration for conceptual insight generation in theory-driven research.
• To investigate how discrepancies between human and LLM coding can be systematically analyzed to derive new theoretical insights.
• To validate the framework through an application to the attribution model of mental illness stigma.
• To demonstrate that LLMs can contribute meaningfully to deeper conceptualization, not just efficiency, in qualitative research.

Proposed method

• LLM-supported data collection to enhance self-disclosure and multi-dimensional data gathering.
• Dual deductive coding by both humans and LLMs using predefined coding schemes to identify disagreement cases.
• Collaborative inductive coding on disagreement cases to explore divergent interpretations and derive new conceptual insights.
• Systematic qualitative analysis of discrepancies between human and LLM coders to uncover latent themes and theoretical implications.
• Integration of LLMs not as passive tools but as active contributors to conceptual development through linguistic nuance detection and argument pattern recognition.
• Application of the CHALET framework to a real-world case study on mental illness stigma to validate its effectiveness.

Experimental results

Research questions

• RQ1 How can human-LLM collaboration be structured to go beyond coding accuracy and support deeper conceptual development in qualitative research?
• RQ2 What types of disagreements between human and LLM coders reveal meaningful theoretical insights when analyzed inductively?
• RQ3 In what ways can LLMs contribute to the discovery of implicit stigmatization themes in qualitative data?
• RQ4 How does the CHALET framework enhance the rigor and depth of theory-driven qualitative analysis in social science research?
• RQ5 What are the implications of LLM-assisted collaborative coding for the future of qualitative methodology in HCI and related fields?

Key findings

• CHALET successfully identified three overarching themes of stigmatization—cognitive, emotional, and behavioral—across participants' narratives on mental illness.
• The framework uncovered sub-themes such as internalized stigma, embarrassment, compassion fatigue, differential support, paternalism, and condescension, revealing nuanced emotional and behavioral responses.
• Disagreements between human and LLM coders were not merely errors but rich sources of insight, especially in detecting emotional and linguistic subtleties related to stigma.
• LLMs demonstrated the ability to detect latent arguments and emotional nuances in narratives, contributing to the identification of previously unarticulated stigmatization mechanisms.
• The collaborative inductive coding process on disagreement cases led to the emergence of new conceptual insights, particularly around the power dynamics in social interactions involving people with mental illness.
• The study demonstrates that LLMs, when integrated through a structured human-LLM synergy model, can significantly enhance the depth and theoretical richness of qualitative analysis.