[Paper Review] Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case

Kaushik Roy, Harshul Surana+5 more|arXiv (Cornell University)|Jan 2, 2025

Mental Health via Writing3 citations

TL;DR

The paper evaluates prompting and fine-tuning LLMs to assist with PHQ-9 and GAD-7 diagnostic assessments, comparing proprietary and open-source models to expert ground truth, and releasing a DiagnosticLlama model and related datasets.

ABSTRACT

Large language models (LLMs) are increasingly attracting the attention of healthcare professionals for their potential to assist in diagnostic assessments, which could alleviate the strain on the healthcare system caused by a high patient load and a shortage of providers. For LLMs to be effective in supporting diagnostic assessments, it is essential that they closely replicate the standard diagnostic procedures used by clinicians. In this paper, we specifically examine the diagnostic assessment processes described in the Patient Health Questionnaire-9 (PHQ-9) for major depressive disorder (MDD) and the Generalized Anxiety Disorder-7 (GAD-7) questionnaire for generalized anxiety disorder (GAD). We investigate various prompting and fine-tuning techniques to guide both proprietary and open-source LLMs in adhering to these processes, and we evaluate the agreement between LLM-generated diagnostic outcomes and expert-validated ground truth. For fine-tuning, we utilize the Mentalllama and Llama models, while for prompting, we experiment with proprietary models like GPT-3.5 and GPT-4o, as well as open-source models such as llama-3.1-8b and mixtral-8x7b.

Motivation & Objective

• Assess whether LLMs can follow standardized PHQ-9 and GAD-7 diagnostic procedures.
• Compare prompting versus fine-tuning approaches across proprietary and open-source models.
• Develop and evaluate a specialized DiagnosticLlama model fine-tuned on diagnostic criteria.
• Create and release clinician-annotated synthetic data and model artifacts to support research.

Proposed method

• Use PHQ-9 and GAD-7 ground-truth datasets from PRIMATE for model guidance.
• Evaluate LLM outputs with hits@k and standard classification metrics (accuracy, precision, recall, F1).
• Explore prompting (naive, exemplar-based, guidance-based) and fine-tuning (SFT, RLHF, DPO) across models.
• Fine-tune MentalllaMa to create DiagnosticLlama and compare to prompting results.
• Release DiagnosticLlama and annotated datasets via Hugging Face and GitHub.

Figure 1. Mental Health Diagnostic Assessment Questionnaires. The Patient Health Questionnaire (PHQ)-9 for depression assessment and the Generalized Anxiety Disorder (GAD)-7 for anxiety assessment.

Experimental results

Research questions

• RQ1 Can LLMs identify PHQ-9 and GAD-7 symptom criteria from posts to match expert ground truth?
• RQ2 How do prompting and fine-tuning impact alignment with clinician assessments?
• RQ3 How do proprietary versus open-source models compare in adherence to diagnostic criteria?
• RQ4 What are the practical limitations and data requirements for reliable LLM-assisted mental health diagnostics?

Key findings

• LLMs approach expert annotation quality for PHQ-9 and GAD-7 tasks in both prompting and fine-tuning setups.
• GPT-4o-mini and mixtral-8x7b excel among proprietary and open-source models, respectively, in key evaluations.
• Fine-tuning a DiagnosticLlama model shows promising results, though fine-tuning is resource-intensive and challenging for this task.
• Older LLMs and non-autoregressive models display noticeable performance gaps compared to modern LLMs.
• Few-shot prompting and fine-tuning generally improve performance relative to zero-shot baselines.