[论文解读] MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models

Kailai Yang, Tianlin Zhang+4 more|arXiv (Cornell University)|Sep 24, 2023

Mental Health via Writing被引用 16

一句话总结

论文介绍了 IMHI，一个用于在社交媒体上进行可解释的心理健康分析的 105k 样本多任务数据集，并训练开源的 MentaLLaMA 指令跟随型大语言模型（LLMs），它们生成具有类人类解释的预测，达到近乎最先进的准确性并具备强大的任务泛化能力。

ABSTRACT

With the development of web technology, social media texts are becoming a rich source for automatic mental health analysis. As traditional discriminative methods bear the problem of low interpretability, the recent large language models have been explored for interpretable mental health analysis on social media, which aims to provide detailed explanations along with predictions. The results show that ChatGPT can generate approaching-human explanations for its correct classifications. However, LLMs still achieve unsatisfactory classification performance in a zero-shot/few-shot manner. Domain-specific finetuning is an effective solution, but faces 2 challenges: 1) lack of high-quality training data. 2) no open-source LLMs for interpretable mental health analysis were released to lower the finetuning cost. To alleviate these problems, we build the first multi-task and multi-source interpretable mental health instruction (IMHI) dataset on social media, with 105K data samples. The raw social media data are collected from 10 existing sources covering 8 mental health analysis tasks. We use expert-written few-shot prompts and collected labels to prompt ChatGPT and obtain explanations from its responses. To ensure the reliability of the explanations, we perform strict automatic and human evaluations on the correctness, consistency, and quality of generated data. Based on the IMHI dataset and LLaMA2 foundation models, we train MentalLLaMA, the first open-source LLM series for interpretable mental health analysis with instruction-following capability. We also evaluate the performance of MentalLLaMA on the IMHI evaluation benchmark with 10 test sets, where their correctness for making predictions and the quality of explanations are examined. The results show that MentalLLaMA approaches state-of-the-art discriminative methods in correctness and generates high-quality explanations.

研究动机与目标

• 将可解释的心理健康分析形式化定义为一个生成任务，该任务产出预测和解释。
• 构建 IMHI，首个用于可解释心理健康分析的多任务、多来源指令微调数据集。
• 开发开源的 MentaLLaMA 模型（7B 和 13B 两个变体），并在一个综合基准上评估它们的正确性和解释质量。
• 展示对未见任务的强泛化能力，并与判别型基线和闭源/开源大型语言模型进行比较。

提出的方法

• 用来自 10 个来源的 8 个心理健康任务的输入-输出对来形式化基于生成的任务。
• 通过将原始帖子、注释和 ChatGPT 生成的解释转换为基于指令的问答对来构建 IMHI 数据集。
• 用领域提示促使专家设计的少-shot 解释，以生成每个注释的高质量解释。
• 使用指令微调和 RLHF，对基于 LLaMA2 的模型（7B 和 13B）进行微调，以构建 MentaLLaMA 变体（包括面向聊天优化的版本）。
• 使用 IMHI 基准进行评估，指标包括预测正确性（加权 F1）和解释质量（BART-score），通过基于 MentalBERT 的一致性检查实现。

Figure 1. Some examples of MentaLLaMA’s capabilities in diverse mental health analysis tasks.

实验结果

研究问题

• RQ1 与判别基线相比，开源指令跟随型 LLM 在可解释的心理健康分析中的表现如何？
• RQ2 指令微调的 LLaMA2 模型是否产生与人类判断一致的高质量解释？
• RQ3 MentaLLaMA 模型在 IMHI 基准内对未见任务的泛化能力如何？

主要发现

• MentaLLaMA-chat-13B 在预测正确性方面，在 10 个测试集中的 7 个接近最先进的判别方法。
• MentaLLaMA 模型生成的解释与 ChatGPT 不相上下，在解释质量方面优于生成式语言模型（PLMs）。
• 指令微调和 RLHF，结合模型规模的增加，提升了跨任务的正确性和解释质量。
• IMHI 数据集使对未见任务具有强泛化能力，在若干设置上表现优于 ChatGPT。

Figure 2. Three components are concatenated to construct the prompts. The key information is marked in blue.