[Paper Review] CoAIcoder: Examining the Effectiveness of AI-assisted Human-to-Human Collaboration in Qualitative Analysis

Jie Gao, Kenny Tsu Wei Choo+3 more|arXiv (Cornell University)|Apr 12, 2023

Qualitative Research Methods and Applications10 citations

TL;DR

CoAIcoder evaluates AI-assisted collaborative qualitative analysis with 32 coder pairs across four collaboration modes, finding that a shared AI model can boost early efficiency and agreement but may reduce code diversity, with independence level shaping outcomes.

ABSTRACT

While AI-assisted individual qualitative analysis has been substantially studied, AI-assisted collaborative qualitative analysis (CQA)-a process that involves multiple researchers working together to interpret data-remains relatively unexplored. After identifying CQA practices and design opportunities through formative interviews, we designed and implemented CoAIcoder, a tool leveraging AI to enhance human-to-human collaboration within CQA through four distinct collaboration methods. With a between-subject design, we evaluated CoAIcoder with 32 pairs of CQA-trained participants across common CQA phases under each collaboration method. Our findings suggest that while using a shared AI model as a mediator among coders could improve CQA efficiency and foster agreement more quickly in the early coding stage, it might affect the final code diversity. We also emphasize the need to consider the independence level when using AI to assist human-to-human collaboration in various CQA scenarios. Lastly, we suggest design implications for future AI-assisted CQA systems.

Motivation & Objective

• Understand how AI can mediate human-to-human collaboration in qualitative analysis (CQA).
• Design and implement CoAIcoder and its four collaboration methods.
• Evaluate the impact on coding efficiency, inter-rater reliability (IRR), and code diversity.
• Derive design implications for future AI-assisted CQA systems.

Proposed method

• Conduct formative semi-structured interviews with QA/CQA researchers to identify practices and needs.
• Develop CoAIcoder using a Rasa NLU pipeline with a pre-trained embedding and real-time fine-tuning on coding history.
• Define four collaboration conditions: Without AI asynchronous without shared model; With AI asynchronous without shared model; With AI asynchronous with shared model; With AI synchronous with shared model.
• Evaluate the four methods in a between-subject design with 32 pairs of CQA-trained participants (8 pairs per method).
• Assess trade-offs between coding efficiency, initial IRR, and code diversity, considering coder independence.

Experimental results

Research questions

• RQ1 Can AI act as an effective mediator to improve efficiency in the initial coding stage of CQA?
• RQ2 How does using a shared AI model influence agreement and final code diversity among coders?
• RQ3 What is the role of coder independence in AI-assisted CQA outcomes?
• RQ4 What design implications emerge for future AI-supported collaborative qualitative analysis tools?

Key findings

• Using a shared AI model as a mediator can improve coding efficiency and initial inter-rater reliability in the early coding stage.
• However, shared AI mediation may reduce final code diversity.
• The level of coder independence modulates the trade-off between efficiency/IRR and code diversity.
• Context matters: the value of efficiency versus code diversity depends on the goals of the CQA study.