[论文解读] Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data

Xuhai Xu, Bingsheng Yao+7 more|arXiv (Cornell University)|Jul 26, 2023

Mental Health via Writing参考文献 122被引用 59

一句话总结

本论文在零-shot、少量样本和指令微调设置下评估多种经过指令微调的LLM在使用在线文本数据的心理健康预测任务中的表现，发现指令微调显著提升性能，Mental-Alpaca 与 Mental-FLAN-T5 达到近似状态-of-the-art 的能力。

ABSTRACT

Advances in large language models (LLMs) have empowered a variety of applications. However, there is still a significant gap in research when it comes to understanding and enhancing the capabilities of LLMs in the field of mental health. In this work, we present a comprehensive evaluation of multiple LLMs on various mental health prediction tasks via online text data, including Alpaca, Alpaca-LoRA, FLAN-T5, GPT-3.5, and GPT-4. We conduct a broad range of experiments, covering zero-shot prompting, few-shot prompting, and instruction fine-tuning. The results indicate a promising yet limited performance of LLMs with zero-shot and few-shot prompt designs for mental health tasks. More importantly, our experiments show that instruction finetuning can significantly boost the performance of LLMs for all tasks simultaneously. Our best-finetuned models, Mental-Alpaca and Mental-FLAN-T5, outperform the best prompt design of GPT-3.5 (25 and 15 times bigger) by 10.9% on balanced accuracy and the best of GPT-4 (250 and 150 times bigger) by 4.8%. They further perform on par with the state-of-the-art task-specific language model. We also conduct an exploratory case study on LLMs' capability on mental health reasoning tasks, illustrating the promising capability of certain models such as GPT-4. We summarize our findings into a set of action guidelines for potential methods to enhance LLMs' capability for mental health tasks. Meanwhile, we also emphasize the important limitations before achieving deployability in real-world mental health settings, such as known racial and gender bias. We highlight the important ethical risks accompanying this line of research.

研究动机与目标

• 评估零-shot、少量样本和指令微调如何影响基于在线文本的心理健康任务上LLM的表现。
• 在多样的数据集上比较一组广泛的LLM（Alpaca, Alpaca-LoRA, FLAN-T5, LLaMA2, GPT-3.5, GPT-4）。
• 证明指令微调在跨任务和数据集上的广泛性能提升。
• 提供开源的面向心理健康的定制模型以及未来研究指南。

提出的方法

• 设计具有四部分结构的零-shot提示以及四种提示策略（Basic, Context Enhancement, Mental Health Enhancement, Context & Mental Health Enhancement）。
• 通过向零-shot提示添加一组提示-标签示例来实现少-shot提示。
• 使用多数据集训练进行指令微调，以实现跨不同心理健康任务的多任务处理。
• 在六个任务上进行评估，这些任务来自七个Reddit与非Reddit数据集，具有训练/测试分割和外部评估。
• 开发并发布 Mental-Alpaca 与 Mental-FLAN-T5 作为开源的心理健康预测任务的微调模型。

Figure 1 . Balanced Accuracy Performance Summary of Mental-Alpaca Finetuning with Different Sizes of Training Set. The finetuning is conducted across four datasets and six tasks. Each solid line represents the performance of the finetuned model on each task. The dashed line indicates the Alpaca ZS p

实验结果

研究问题

• RQ1 零-shot 和少-shot 提示在多种 LLM 上的心理健康预测任务中有何比较？
• RQ2 指令微调在多样的心理健康任务和数据集上能在多大程度上提升性能？
• RQ3 微调模型能否匹配或超过面向任务的模型在心理健康预测中的表现？
• RQ4 在将 LLMs 应用于心理健康分析时有哪些伦理考量和偏见？

主要发现

• 指令微调显著提升多种心理健康任务与数据集上的模型性能。
• Mental-Alpaca 与 Mental-FLAN-T5 在平均平衡准确率上比最佳的 GPT-3.5 提示设计高出 10.9%（尽管模型要小得多），并且在平均上比最佳的 GPT-4 提示高出 4.8%。
• 微调模型在若干任务上达到与最新的特定任务模型 Mental-RoBERTa 的能力相当。
• 零-shot 和少-shot 提示显示出有前景但有限的性能，少-shot 提供的平均提升约为 4.1%。
• 案例研究显示在 GPT-4 这样的模型中存在细致的推理能力，并突出需要进一步研究的失败案例。
• 开源发布 Mental-Alpaca 和 Mental-FLAN-T5，以促进多任务心理健康预测的研究。

Figure 2 . A Case Study of Correct Reasoning Examples on Task #1 Binary Stress Prediction on Dreaddit Dataset. Bolded texts highlight the mental-health-related content in the input section, and the answers of LLMs. Underlined texts highlight the reasoning content generated by LLMs, and italicized &