[Paper Review] Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data

Xuhai Xu, Bingsheng Yao+7 more|arXiv (Cornell University)|Jul 26, 2023

Mental Health via Writing122 references59 citations

TL;DR

The paper evaluates multiple instruction-tuned LLMs across zero-shot, few-shot, and instruction-finetuning setups for mental health prediction tasks using online text data, finding that instruction finetuning substantially boosts performance and that Mental-Alpaca and Mental-FLAN-T5 reach state-of-the-art-like capabilities.

ABSTRACT

Advances in large language models (LLMs) have empowered a variety of applications. However, there is still a significant gap in research when it comes to understanding and enhancing the capabilities of LLMs in the field of mental health. In this work, we present a comprehensive evaluation of multiple LLMs on various mental health prediction tasks via online text data, including Alpaca, Alpaca-LoRA, FLAN-T5, GPT-3.5, and GPT-4. We conduct a broad range of experiments, covering zero-shot prompting, few-shot prompting, and instruction fine-tuning. The results indicate a promising yet limited performance of LLMs with zero-shot and few-shot prompt designs for mental health tasks. More importantly, our experiments show that instruction finetuning can significantly boost the performance of LLMs for all tasks simultaneously. Our best-finetuned models, Mental-Alpaca and Mental-FLAN-T5, outperform the best prompt design of GPT-3.5 (25 and 15 times bigger) by 10.9% on balanced accuracy and the best of GPT-4 (250 and 150 times bigger) by 4.8%. They further perform on par with the state-of-the-art task-specific language model. We also conduct an exploratory case study on LLMs' capability on mental health reasoning tasks, illustrating the promising capability of certain models such as GPT-4. We summarize our findings into a set of action guidelines for potential methods to enhance LLMs' capability for mental health tasks. Meanwhile, we also emphasize the important limitations before achieving deployability in real-world mental health settings, such as known racial and gender bias. We highlight the important ethical risks accompanying this line of research.

Motivation & Objective

• Assess how zero-shot, few-shot, and instruction finetuning affect LLMs on mental health tasks using online text data.
• Compare a wide set of LLMs (Alpaca, Alpaca-LoRA, FLAN-T5, LLaMA2, GPT-3.5, GPT-4) on diverse datasets.
• Demonstrate that instruction finetuning yields broad performance gains across tasks and datasets.
• Provide open-source mental-health tailored models and guidelines for future research.

Proposed method

• Design zero-shot prompts with four-part structure and four prompt strategies (Basic, Context Enhancement, Mental Health Enhancement, Context & Mental Health Enhancement).
• Implement few-shot prompting by adding a set of prompt-label demonstrations to the zero-shot prompt.
• Perform instruction finetuning using multi-dataset training to enable multi-task handling across different mental-health tasks.
• Evaluate on six tasks drawn from seven Reddit and non-Reddit datasets with train/test splits and external evaluations.
• Develop and release Mental-Alpaca and Mental-FLAN-T5 as open-source finetuned models for mental-health prediction tasks.

Figure 1 . Balanced Accuracy Performance Summary of Mental-Alpaca Finetuning with Different Sizes of Training Set. The finetuning is conducted across four datasets and six tasks. Each solid line represents the performance of the finetuned model on each task. The dashed line indicates the Alpaca ZS p

Experimental results

Research questions

• RQ1 How do zero-shot and few-shot prompting compare across multiple LLMs on mental health prediction tasks?
• RQ2 To what extent can instruction finetuning improve performance across diverse mental health tasks and datasets?
• RQ3 Can finetuned models match or surpass task-specific models in mental health prediction?
• RQ4 What are the ethical considerations and biases when deploying LLMs for mental health analyses?

Key findings

• Instruction finetuning significantly boosts model performance across multiple mental-health tasks and datasets.
• Mental-Alpaca and Mental-FLAN-T5 outperform the best GPT-3.5 prompt designs by 10.9% average balance accuracy (despite being much smaller) and beat the best GPT-4 prompts by 4.8% on average.
• Finetuned models reach parity with the state-of-the-art task-specific Mental-RoBERTa on several tasks.
• Zero-shot and few-shot prompts show promising but limited performance, with few-shot giving an average improvement of about 4.1%.
• Case study indicates nuanced reasoning capabilities in models like GPT-4 and highlights failure cases requiring further research.
• Open-source release of Mental-Alpaca and Mental-FLAN-T5 to facilitate research in multi-task mental-health prediction.

Figure 2 . A Case Study of Correct Reasoning Examples on Task #1 Binary Stress Prediction on Dreaddit Dataset. Bolded texts highlight the mental-health-related content in the input section, and the answers of LLMs. Underlined texts highlight the reasoning content generated by LLMs, and italicized &