[Paper Review] MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models

Kailai Yang, Tianlin Zhang+4 more|arXiv (Cornell University)|Sep 24, 2023

Mental Health via WritingPsychology16 citations

TL;DR

The paper introduces IMHI, a 105k-sample multi-task dataset for interpretable mental health analysis on social media, and trains open-source MentaLLaMA instruction-following LLMs that generate predictions with human-like explanations, achieving near state-of-the-art accuracy and strong task generalization.

ABSTRACT

With the development of web technology, social media texts are becoming a rich source for automatic mental health analysis. As traditional discriminative methods bear the problem of low interpretability, the recent large language models have been explored for interpretable mental health analysis on social media, which aims to provide detailed explanations along with predictions. The results show that ChatGPT can generate approaching-human explanations for its correct classifications. However, LLMs still achieve unsatisfactory classification performance in a zero-shot/few-shot manner. Domain-specific finetuning is an effective solution, but faces 2 challenges: 1) lack of high-quality training data. 2) no open-source LLMs for interpretable mental health analysis were released to lower the finetuning cost. To alleviate these problems, we build the first multi-task and multi-source interpretable mental health instruction (IMHI) dataset on social media, with 105K data samples. The raw social media data are collected from 10 existing sources covering 8 mental health analysis tasks. We use expert-written few-shot prompts and collected labels to prompt ChatGPT and obtain explanations from its responses. To ensure the reliability of the explanations, we perform strict automatic and human evaluations on the correctness, consistency, and quality of generated data. Based on the IMHI dataset and LLaMA2 foundation models, we train MentalLLaMA, the first open-source LLM series for interpretable mental health analysis with instruction-following capability. We also evaluate the performance of MentalLLaMA on the IMHI evaluation benchmark with 10 test sets, where their correctness for making predictions and the quality of explanations are examined. The results show that MentalLLaMA approaches state-of-the-art discriminative methods in correctness and generates high-quality explanations.

Motivation & Objective

• Formally define interpretable mental health analysis as a generation task that yields predictions and explanations.
• Build IMHI, the first multi-task, multi-source instruction-tuning dataset for interpretable mental health analysis.
• Develop open-source MentaLLaMA models (7B and 13B variants) and assess their correctness and explanation quality on a holistic benchmark.
• Demonstrate strong generalization to unseen tasks and compare against discriminative baselines and closed/open LLMs.

Proposed method

• Formalize the generation-based task with input-output pairs across 8 mental health tasks collected from 10 sources.
• Construct the IMHI dataset by transforming raw posts, annotations, and ChatGPT-generated explanations into instruction-based Q-A pairs.
• Prompt expert-designed few-shot explanations with domain prompts to generate high-quality explanations for each annotation.
• Fine-tune LLaMA2-based models (7B and 13B) with instruction tuning and RLHF to build MentaLLaMA variants (including chat-optimized versions).
• Evaluate using the IMHI benchmark with metrics for prediction correctness (weighted F1) and explanation quality (BART-score) via MentalBERT-based consistency checks.

Figure 1. Some examples of MentaLLaMA’s capabilities in diverse mental health analysis tasks.

Experimental results

Research questions

• RQ1 How well can open-source instruction-following LLMs perform on interpretable mental health analysis compared to discriminative baselines?
• RQ2 Do instruction-tuned LLaMA2 models produce high-quality explanations that align with human judgments?
• RQ3 How well do the MentaLLaMA models generalize to unseen tasks within the IMHI benchmark?

Key findings

• MentaLLaMA-chat-13B approaches state-of-the-art discriminative methods in 7 of 10 test sets for prediction correctness.
• MentaLLaMA models generate explanations on par with ChatGPT and outperform generative PLMs in explanation quality.
• Instruction tuning and RLHF, along with increased model size, improve correctness and explanation quality across tasks.
• The IMHI dataset enables strong generalization to unseen tasks, with better performance than ChatGPT on several settings.

Figure 2. Three components are concatenated to construct the prompts. The key information is marked in blue.