[Paper Review] Helping the Helper: Supporting Peer Counselors via AI-Empowered Practice and Feedback

Shang-Ling Hsu, Raj Sanjay Shah+5 more|arXiv (Cornell University)|May 15, 2023

Digital Mental Health Interventions10 citations

TL;DR

CARE is an interactive AI tool that assists peer counselors by predicting suitable Motivational Interviewing strategies and generating example responses in real time, evaluated through quantitative logs and qualitative user studies.

ABSTRACT

Millions of users come to online peer counseling platforms to seek support. However, studies show that online peer support groups are not always as effective as expected, largely due to users' negative experiences with unhelpful counselors. Peer counselors are key to the success of online peer counseling platforms, but most often do not receive appropriate training.Hence, we introduce CARE: an AI-based tool to empower and train peer counselors through practice and feedback. Concretely, CARE helps diagnose which counseling strategies are needed in a given situation and suggests example responses to counselors during their practice sessions. Building upon the Motivational Interviewing framework, CARE utilizes large-scale counseling conversation data with text generation techniques to enable these functionalities. We demonstrate the efficacy of CARE by performing quantitative evaluations and qualitative user studies through simulated chats and semi-structured interviews, finding that CARE especially helps novice counselors in challenging situations. The code is available at https://github.com/SALT-NLP/CARE

Motivation & Objective

• Address the challenge of scaling effective training for online peer counselors.
• Provide real-time, context-aware AI assistance to novices without replacing human counselors.
• Ground the system in Motivational Interviewing (MI) strategies and large-scale counseling data.
• Demonstrate the efficacy of CARE through quantitative system logs and qualitative user studies.

Proposed method

• Train binary classifiers (BERT-based) to predict the most suitable MI strategy from the five most recent utterances in a chat.
• Condition a DialoGPT-based generator on the predicted MI strategy to produce up to three example responses per situation.
• Filter generated responses with a HateBERT-based inappropriate content classifier to ensure safety.
• Integrate the backend models with a front-end interface modeled after 7 Cups to enable real-time suggestions during training chats.
• Utilize 7 Cups datasets (7C-HQ, 7C-MI) for training, validation, and evaluation; evaluate via system logs, questionnaires, and semi-structured interviews.

Figure 1 . Example interface of CARE to empower supporters by (1) diagnosing which skill to use and (2) suggesting responses.

Experimental results

Research questions

• RQ1 Does CARE improve the quality and usefulness of peer counselor responses in simulated training chats?
• RQ2 How accurately can BERT-based classifiers predict appropriate MI strategies in given conversation contexts?
• RQ3 How do strategy-conditioned generation models (DialoGPT-2, GPT-2, BART) perform in producing appropriate and safe example responses?
• RQ4 What is the perceived usefulness and usability of CARE among novice and experienced peer counselors?

Key findings

• The MI-strategy classifiers achieve an overall F1 score greater than 0.705 on the test set 7C-MI.
• DialoGPT-2 conditioned on predicted strategies outperforms GPT-2 and BART in generation tasks and achieves a semantic similarity (BERTScore) above 0.87 with about 78% agreement (Positive) with the intended strategy.
• The safety filter using HateBERT achieves an F1 score of 84.21% on the 7C-MI test set, indicating strong recall for inappropriate content.
• Generation quality varies by strategy, with higher positive rates for Grounding, Open Questions, and Introduction, and lower rates for Reflection, Persuade, and Affirm.
• The system is designed for training and simulated chats, with real-time suggestions and full human oversight, ensuring counselor control over utilized content.

Figure 2 . CARE system architecture. CARE consists of 8 next-strategy predictors, one next-response generator, and one inappropriate response classifier. It predicts the probability distribution of the next counseling strategies and generates suggested responses for each confident strategy independe