[Paper Review] Psy-LLM: Scaling up Global Mental Health Psychological Services with AI-based Large Language Models

Tin Lai, Yukun Shi+5 more|arXiv (Cornell University)|Jul 22, 2023

Mental Health via Writing34 citations

TL;DR

Psy-LLM proposes an AI-based front-end tool for online psychological consultation by fine-tuning PanGu and WenZhong on professional Q&A and crawled psychology data, evaluated with perplexity and human ratings, to assist clinicians and screen urgent cases.

ABSTRACT

The demand for psychological counselling has grown significantly in recent years, particularly with the global outbreak of COVID-19, which has heightened the need for timely and professional mental health support. Online psychological counselling has emerged as the predominant mode of providing services in response to this demand. In this study, we propose the Psy-LLM framework, an AI-based assistive tool leveraging Large Language Models (LLMs) for question-answering in psychological consultation settings to ease the demand for mental health professions. Our framework combines pre-trained LLMs with real-world professional Q\&A from psychologists and extensively crawled psychological articles. The Psy-LLM framework serves as a front-end tool for healthcare professionals, allowing them to provide immediate responses and mindfulness activities to alleviate patient stress. Additionally, it functions as a screening tool to identify urgent cases requiring further assistance. We evaluated the framework using intrinsic metrics, such as perplexity, and extrinsic evaluation metrics, with human participant assessments of response helpfulness, fluency, relevance, and logic. The results demonstrate the effectiveness of the Psy-LLM framework in generating coherent and relevant answers to psychological questions. This article discusses the potential and limitations of using large language models to enhance mental health support through AI technologies.

Motivation & Objective

• Address the shortage of licensed mental health professionals by enabling AI-assisted online psychological consultation.
• Develop a frontline tool for professionals to provide instant responses and mindfulness activities, and to screen urgent cases.
• Combine pre-trained Chinese LLMs with domain-specific Q&A data to improve counselling quality and accessibility.
• Evaluate the framework using intrinsic (perplexity) and extrinsic (human ratings) metrics and real-world deployment feedback.

Proposed method

• Leverage PanGu and WenZhong large-scale pre-trained Chinese language models as the base.
• Fine-tune on PsyQA (22,000 questions and 56,000 answers) and extensive crawled Chinese psychological articles.
• Collect and clean data (remove duplicates, ads, short samples, URLs, usernames, and punctuation normalization) to create a high-quality training corpus.
• Train PanGu 350M on a 2.85GB psychology corpus and fine-tune with PsyQA data; use 100,000 iterations for convergence.
• Incorporate WenZhong-110M for additional fine-tuning and adapt data processing to model requirements (tokenization, max sequence length).
• Evaluate dataset quality via perplexity and expert psychology assessment; deploy a dedicated website for user feedback and iterative refinement.

Experimental results

Research questions

• RQ1 How effectively can Psy-LLM generate coherent, relevant, and professional psychological responses in an online consultation context?
• RQ2 Can Psy-LLM assist human counsellors by reducing workload and aiding urgent-case screening during high-demand periods?
• RQ3 What is the comparative impact of PanGu versus WenZhong as base models for psychology-focused QA in Chinese?
• RQ4 Does a web-based frontend with AI-assisted responses improve accessibility and reduce stigma in seeking mental health support?

Key findings

• The framework can generate responses to users within seconds when deployed on a server.
• A data pipeline combining PsyQA and crawled Chinese psychology data yields a training corpus used for fine-tuning PanGu 350M and WenZhong-110M.
• Perplexity-based data quality evaluation complemented by expert psychologist review guided data cleaning and model training.
• The dataset comprises about 400,000 samples, with most data sourced from Tianya (about 2GB), plus Zhihu and Yixinli contributions, reflecting domain-relevant content.
• The approach targets both clinician-support (assistive tool) and patient-facing online consultation during times when human counsellors are unavailable.
• The study discusses potential benefits and limitations of applying large language models to mental health support, including ethical and reliability considerations.