[Paper Review] An Assessment on Comprehending Mental Health through Large Language Models

Mihael Arčan, Paul-David Niland+1 more|arXiv (Cornell University)|Jan 9, 2024

Mental Health via Writing4 citations

TL;DR

This study evaluates large language models (LLMs) like Llama-2 and ChatGPT against classical transformer models (e.g., Distil-RoBERTa, BERT) for mental health symptom detection using the DAIC-WOZ dataset. Despite their scale, LLMs underperform compared to fine-tuned, smaller transformer models, with Distil-RoBERTa achieving the best performance across GAD and PHQ question sets in precision, recall, and F1 scores.

ABSTRACT

Mental health challenges pose considerable global burdens on individuals and communities. Recent data indicates that more than 20% of adults may encounter at least one mental disorder in their lifetime. On the one hand, the advancements in large language models have facilitated diverse applications, yet a significant research gap persists in understanding and enhancing the potential of large language models within the domain of mental health. On the other hand, across various applications, an outstanding question involves the capacity of large language models to comprehend expressions of human mental health conditions in natural language. This study presents an initial evaluation of large language models in addressing this gap. Due to this, we compare the performance of Llama-2 and ChatGPT with classical Machine as well as Deep learning models. Our results on the DAIC-WOZ dataset show that transformer-based models, like BERT or XLNet, outperform the large language models.

Motivation & Objective

• To evaluate the capability of large language models (LLMs) such as Llama-2 and ChatGPT in comprehending and detecting mental health conditions from natural language.
• To compare the performance of LLMs against classical machine learning and deep learning models, particularly fine-tuned transformer architectures, in mental health symptom detection.
• To investigate whether prompting strategies improve LLM performance on mental health question sets (GAD-1, GAD-2, PHQ-1, PHQ-2).
• To identify key performance gaps and biases in LLMs when applied to sensitive mental health data, especially given the high stakes of misdiagnosis or harmful advice.
• To provide empirical benchmarks for future development of AI tools in mental health, emphasizing the need for model adaptation and bias mitigation.

Proposed method

• Fine-tuned transformer models including BERT, RoBERTa, XLNet, and Distil-RoBERTa were trained and evaluated on the DAIC-WOZ dataset for mental health symptom detection.
• Large language models Llama-2 and ChatGPT were prompted using multiple strategies (e.g., few-shot prompting, zero-shot prompting) to classify symptoms from clinical dialogue transcripts.
• Model performance was evaluated using standard NLP metrics: weighted precision, recall, F1-score, Hamming loss, and AUC-ROC for multi-label classification.
• XGBoost was used as a strong baseline model to compare with deep learning and LLM approaches, leveraging hand-crafted features and gradient boosting.
• The study employed a zero-shot and few-shot prompting framework for LLMs, assessing their few-shot generalization ability on mental health symptom detection.
• All models were evaluated on four mental health question sets: GAD-1, GAD-2 (generalized anxiety disorder), PHQ-1, and PHQ-2 (depression symptoms), using the PHQ-4 scale as reference.

Experimental results

Research questions

• RQ1 Can large language models like Llama-2 and ChatGPT effectively detect mental health symptoms from clinical dialogue transcripts without fine-tuning?
• RQ2 How do LLMs compare in performance to fine-tuned, smaller transformer models (e.g., Distil-RoBERTa, BERT) on mental health symptom detection tasks?
• RQ3 Does prompting strategy significantly improve the performance of LLMs on mental health classification tasks compared to standard fine-tuning?
• RQ4 What is the relative performance of XGBoost and deep learning models versus LLMs in terms of Hamming loss and AUC-ROC for multi-label mental health prediction?
• RQ5 To what extent do LLMs exhibit bias or inconsistency in detecting symptoms of anxiety and depression, especially given the sensitivity of mental health data?

Key findings

• Distil-RoBERTa achieved the highest weighted F1-score of 0.68 on the GAD-2 question and 0.62 on PHQ-2, outperforming all other models including Llama-2 and ChatGPT.
• ChatGPT 3.5 showed minor advantages over Llama-2 (v3), achieving a higher F1-score of 0.53 on PHQ-2 (0.49 for Llama-2) and better AUC-ROC on GAD-2 (0.77 vs. 0.70).
• Llama-2 (v3) performed poorly across all metrics, with a weighted F1 of only 0.33 on GAD-2 and 0.30 on PHQ-2, indicating limited zero-shot generalization for mental health tasks.
• XGBoost consistently outperformed all models in Hamming loss, achieving 0.31 on GAD-2 and 0.47 on PHQ-2, demonstrating strong performance on multi-label classification.
• Transformer-based models like BERT, XLNet, and RoBERTa significantly outperformed LLMs in all metrics, with Distil-RoBERTa achieving the best balance of performance and efficiency.
• The study found that larger parameter count in LLMs does not translate to better performance in mental health symptom detection, challenging assumptions about LLM generalization in clinical NLP.