[Paper Review] Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support

Ashish Sharma, Inna Wanyin Lin+3 more|arXiv (Cornell University)|Mar 28, 2022

Digital Mental Health Interventions22 citations

TL;DR

The study introduces Hailey, an AI-in-the-loop system that provides just-in-time empathic-writing feedback to peer supporters on TalkLife, yielding substantial increases in expressed empathy in online peer-to-peer mental health conversations.

ABSTRACT

Advances in artificial intelligence (AI) are enabling systems that augment and collaborate with humans to perform simple, mechanistic tasks like scheduling meetings and grammar-checking text. However, such Human-AI collaboration poses challenges for more complex, creative tasks, such as carrying out empathic conversations, due to difficulties of AI systems in understanding complex human emotions and the open-ended nature of these tasks. Here, we focus on peer-to-peer mental health support, a setting in which empathy is critical for success, and examine how AI can collaborate with humans to facilitate peer empathy during textual, online supportive conversations. We develop Hailey, an AI-in-the-loop agent that provides just-in-time feedback to help participants who provide support (peer supporters) respond more empathically to those seeking help (support seekers). We evaluate Hailey in a non-clinical randomized controlled trial with real-world peer supporters on TalkLife (N=300), a large online peer-to-peer support platform. We show that our Human-AI collaboration approach leads to a 19.60% increase in conversational empathy between peers overall. Furthermore, we find a larger 38.88% increase in empathy within the subsample of peer supporters who self-identify as experiencing difficulty providing support. We systematically analyze the Human-AI collaboration patterns and find that peer supporters are able to use the AI feedback both directly and indirectly without becoming overly reliant on AI while reporting improved self-efficacy post-feedback. Our findings demonstrate the potential of feedback-driven, AI-in-the-loop writing systems to empower humans in open-ended, social, creative tasks such as empathic conversations.

Motivation & Objective

• Motivate and enable empathic, open-ended support in online peer-to-peer mental health platforms where many supporters are untrained.
• Develop and evaluate an AI-in-the-loop feedback agent that can provide actionable, just-in-time empathy guidance to peer supporters.
• Assess whether Human-AI collaboration increases expressed empathy beyond traditional training alone in a non-clinical randomized trial.

Proposed method

• Design Hailey to offer just-in-time feedback with Insert and Replace options based on the seeker post and current supporter response.
• Conduct a non-clinical randomized controlled trial on TalkLife with N=300 participants comparing Human+AI (treatment) vs Human Only (control).
• Use both human evaluation and automatic empathy scoring to assess outcomes, ensuring safety with post-hoc safety and ethics measures.

Experimental results

Research questions

• RQ1 Does just-in-time Human-AI feedback increase empathic responses in peer supporters compared to no feedback?
• RQ2 How does Human-AI collaboration affect supporters with difficulty writing empathic responses vs those without such difficulties?
• RQ3 What are the patterns of Human-AI collaboration (consultation frequency and usage) and their relation to empathy gains?

Key findings

• Human+AI feedback yields a 19.60% higher empathic scores versus Human Only (1.77 vs. 1.48 on a 0–6 scale, p<1e-5).
• Independent human evaluations rate Human+AI responses as more empathic 46.87% of the time vs 37.40% for Human Only (p<0.01).
• Among participants reporting writing challenges, empathy gains were 38.88% for Human+AI versus 11.87% for Human Only (p<1e-5).
• Participants who consulted AI more often showed higher expressed empathy, with frequent AI use correlating with higher empathy scores.
• 63.31% found the feedback helpful, 60.43% found it actionable, and 77.70% wanted deployment on platforms like TalkLife.
• 69.78% reported feeling more confident providing support after the study.