[Paper Review] Multi-dimensional Assessment and Explainable Feedback for Counselor Responses to Client Resistance in Text-based Counseling with LLMs

Anqi Li, Ruihan Wang+6 more|arXiv (Cornell University)|Feb 25, 2026

Mental Health via Writing0 citations

TL;DR

The paper proposes a four-dimension framework to evaluate counselor responses to client resistance in text-based counseling, creates an expert-annotated dataset, and trains a fine-tuned Llama-3.1 model that outperforms baselines and can generate explanations; a proof-of-concept study shows AI-generated feedback improves counselors’ response quality.

ABSTRACT

Effectively addressing client resistance is a sophisticated clinical skill in psychological counseling, yet practitioners often lack timely and scalable supervisory feedback to refine their approaches. Although current NLP research has examined overall counseling quality and general therapeutic skills, it fails to provide granular evaluations of high-stakes moments where clients exhibit resistance. In this work, we present a comprehensive pipeline for the multi-dimensional evaluation of human counselors' interventions specifically targeting client resistance in text-based therapy. We introduce a theory-driven framework that decomposes counselor responses into four distinct communication mechanisms. Leveraging this framework, we curate and share an expert-annotated dataset of real-world counseling excerpts, pairing counselor-client interactions with professional ratings and explanatory rationales. Using this data, we perform full-parameter instruction tuning on a Llama-3.1-8B-Instruct backbone to model fine-grained evaluative judgments of response quality and generate explanations underlying. Experimental results show that our approach can effectively distinguish the quality of different communication mechanisms (77-81% F1), substantially outperforming GPT-4o and Claude-3.5-Sonnet (45-59% F1). Moreover, the model produces high-quality explanations that closely align with expert references and receive near-ceiling ratings from human experts (2.8-2.9/3.0). A controlled experiment with 43 counselors further confirms that receiving these AI-generated feedback significantly improves counselors' ability to respond effectively to client resistance.

Motivation & Objective

• Develop a theory-driven, multi-dimensional framework to assess counselor responses to client resistance in text-based counseling.
• Create an expert-annotated dataset with explanations for resistance-response interventions.
• Train a large language model to produce fine-grained evaluations and actionable, interpretable explanations.
• Demonstrate the practical utility of AI-generated feedback to improve counselor performance in resisting contexts.

Proposed method

• Propose a four-dimensional framework: Respect for Autonomy, Stance Alignment, Emotional Resonance, and Conversational Orientation with three expression levels each (no, weak, strong).
• Construct an expert-annotated dataset from ClientBehavior and ObserverWAI dialogues with resistance detection and counselor responses, including explanations.
• Perform full-parameter fine-tuning of Llama-3.1-8B-Instruct on the task with 5-fold cross-validation and oversampling to address class imbalance.
• Evaluate classification performance against baselines (including GPT-4o and Claude-3.5-Sonnet) using macro-F1 and accuracy; assess explanation quality via automatic metrics (BLEU/Rouge) and human ratings.
• Conduct a proof-of-concept study with 43 counselors to test AI-generated feedback efficacy using linear mixed-effects models.

Figure 1: Overview of our framework for evaluating counselor responses to client resistance. The framework comprises four core communication mechanisms: Respect for Autonomy , Stance Alignment , Emotional Resonance , and Conversational Orientation . For each mechanism, responses are further categori

Experimental results

Research questions

• RQ1 Can a multi-dimensional framework reliably distinguish levels of counselor response expressions across four communication mechanisms in resistance contexts?
• RQ2 Does task-specific fine-tuning with explanations improve classification and explanation quality beyond baseline LLMs?
• RQ3 Are AI-generated explanations and feedback usable and beneficial for real-time counselor training and skill development?

Key findings

• Our model achieves macro-F1 77.34–81.00% and accuracy 77.64–87.06% across the four mechanisms, outperforming GPT-4o and Claude-3.5-Sonnet by 20+ F1 points.
• Incorporating explanations during training yields at least ~4 F1-point gains over label-only training.
• Automated explanations achieve BLEU-1 = 0.60 and human-evaluated quality scores of 2.8–2.9/3.0 for framework consistency, evidence anchoring, and clarity/specificity.
• Explanations show strong lexical alignment (BLEU-1 0.60) and near-ceiling human ratings, indicating high-quality, actionable feedback.
• In a controlled trial, counselors receiving AI-generated feedback significantly improved their resistance-response quality vs. a control group (phase interaction effects across all four dimensions).
• Annotation reliability is substantial (Cohen’s kappa 0.74–0.77) across four mechanisms, with high-quality explanatory rationales.

Figure 2: Interaction effects between experimental groups and phases across four dimensions. Solid green lines represent the control group, while dashed orange lines represent the experimental group. Points denote the mean values, and error bars indicate 95% confidence intervals. The results reveal