[Paper Review] Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Tools

Jung In Park, Mahyar Abbasian+12 more|arXiv (Cornell University)|Aug 3, 2024

Digital Mental Health InterventionsPsychology3 citations

TL;DR

This paper proposes a standardized, expert-validated evaluation framework for mental health chatbots using 100 benchmark questions, ideal responses, and guideline-based assessment. It demonstrates that an agentic LLM approach with real-time data access achieves the highest alignment with human evaluations, significantly improving safety and reliability over static LLM scoring methods.

ABSTRACT

Objective: This study aims to develop and validate an evaluation framework to ensure the safety and reliability of mental health chatbots, which are increasingly popular due to their accessibility, human-like interactions, and context-aware support. Materials and Methods: We created an evaluation framework with 100 benchmark questions and ideal responses, and five guideline questions for chatbot responses. This framework, validated by mental health experts, was tested on a GPT-3.5-turbo-based chatbot. Automated evaluation methods explored included large language model (LLM)-based scoring, an agentic approach using real-time data, and embedding models to compare chatbot responses against ground truth standards. Results: The results highlight the importance of guidelines and ground truth for improving LLM evaluation accuracy. The agentic method, dynamically accessing reliable information, demonstrated the best alignment with human assessments. Adherence to a standardized, expert-validated framework significantly enhanced chatbot response safety and reliability. Discussion: Our findings emphasize the need for comprehensive, expert-tailored safety evaluation metrics for mental health chatbots. While LLMs have significant potential, careful implementation is necessary to mitigate risks. The superior performance of the agentic approach underscores the importance of real-time data access in enhancing chatbot reliability. Conclusion: The study validated an evaluation framework for mental health chatbots, proving its effectiveness in improving safety and reliability. Future work should extend evaluations to accuracy, bias, empathy, and privacy to ensure holistic assessment and responsible integration into healthcare. Standardized evaluations will build trust among users and professionals, facilitating broader adoption and improved mental health support through technology.

Motivation & Objective

• Address the growing need for reliable, safe mental health chatbots due to their increasing use in accessible, human-like care.
• Develop a comprehensive evaluation framework to assess chatbot safety, accuracy, and reliability in mental health contexts.
• Validate the framework using mental health experts to ensure clinical relevance and reduce risks of harmful responses.
• Compare automated evaluation methods, particularly LLM-based scoring and agentic approaches, to identify the most effective techniques.
• Establish a foundation for standardized, holistic evaluation of chatbots across safety, empathy, bias, and privacy dimensions.

Proposed method

• Designed a benchmark framework with 100 curated questions and ideal responses, validated by mental health professionals.
• Defined five guideline questions to assess chatbot responses for safety, clinical appropriateness, and ethical alignment.
• Implemented three automated evaluation methods: LLM-based scoring, agentic retrieval with real-time data access, and embedding-based similarity to ground truth.
• Used GPT-3.5-turbo as the base model for chatbot responses, evaluated against expert-validated standards.
• Applied embedding models (e.g., sentence transformers) to compute semantic similarity between chatbot outputs and ideal responses.
• Evaluated performance by comparing automated scores against human assessments from mental health experts.

Experimental results

Research questions

• RQ1 How effective is a standardized, expert-validated evaluation framework in improving the safety and reliability of mental health chatbots?
• RQ2 Which automated evaluation method—LLM-based scoring, agentic retrieval, or embedding similarity—most closely aligns with human expert judgments?
• RQ3 To what extent does real-time data access through an agentic approach enhance the accuracy and safety of chatbot responses?
• RQ4 How does adherence to clinical guidelines and ground truth responses affect LLM evaluation performance?
• RQ5 Can automated evaluation tools reliably assess chatbot responses across key mental health dimensions like safety, empathy, and bias?

Key findings

• The agentic evaluation approach, which dynamically accesses reliable information sources, showed the strongest alignment with human expert assessments.
• Guidelines and ground truth standards significantly improved the accuracy of LLM-based evaluation methods.
• Static LLM scoring without real-time data access performed less reliably than the agentic method, indicating limitations in standalone LLM reasoning.
• The expert-validated framework effectively enhanced response safety and clinical appropriateness across diverse mental health scenarios.
• Embedding-based similarity measures provided useful but less precise alignment with human judgments compared to the agentic approach.
• Standardized evaluation frameworks are essential for building trust and enabling responsible deployment of mental health chatbots in clinical settings.