[Paper Review] Depression Detection on Social Media with Large Language Models

Xiaochong Lan, Han, Zhiguang+5 more|arXiv (Cornell University)|Mar 16, 2024

Mental Health via Writing13 citations

TL;DR

DORIS combines DSM-5 based medical knowledge with large language models and embedding features to detect depression from social media posts, achieving higher AUPRC and providing explanations. It outperforms baselines on the SWDD dataset.

ABSTRACT

Limited access to mental healthcare resources hinders timely depression diagnosis, leading to detrimental outcomes. Social media platforms present a valuable data source for early detection, yet this task faces two significant challenges: 1) the need for medical knowledge to distinguish clinical depression from transient mood changes, and 2) the dual requirement for high accuracy and model explainability. To address this, we propose DORIS, a framework that leverages Large Language Models (LLMs). To integrate medical knowledge, DORIS utilizes LLMs to annotate user texts against established medical diagnostic criteria and to summarize historical posts into temporal mood courses. These medically-informed features are then used to train an accurate Gradient Boosting Tree (GBT) classifier. Explainability is achieved by generating justifications for predictions based on the LLM-derived symptom annotations and mood course analyses. Extensive experimental results validate the effectiveness as well as interpretability of our method, highlighting its potential as a supportive clinical tool.

Motivation & Objective

• Integrate professional medical knowledge (DSM-5) into an automated depression detection system.
• Leverage LLMs to annotate depression symptoms and summarize mood histories.
• Fuse medical-knowledge features with embedding-based representations for accurate, explainable predictions.
• Demonstrate improved performance (AUPRC) and provide interpretable explanations of results.

Proposed method

• Use DSM-5 symptom criteria to create a 9-dimensional per-post depression symptom vector annotated by an LLM.
• Identify high-emotion posts and synthesize mood-course descriptions via LLMs; embed and aggregate into mood-course features.
• Construct three user features: diagnostic-criteria (DC), mood course (MC), and post history (PH).
• Combine features in a gradient-boosting tree (GBT) classifier for final depression prediction.
• Explainability is achieved by having LLMs annotate symptoms and mood-course and generate explanations (T^Exp) for predictions.
• Efficient annotation via filtering to annotate only high-risk posts (top-k similarity) to reduce LLM usage.

Figure 1: Illustration of depression detection on social media. It aims to determine whether a user has depression based on their history of posts on social media.

Experimental results

Research questions

• RQ1 Can DSM-5-based symptom annotation by LLMs improve depression detection on social media?
• RQ2 Does modeling mood course and high-emotion post history improve accuracy and explainability over text-based baselines?
• RQ3 Can a hybrid system combining medical knowledge with LLMs and traditional classifiers outperform pure LLM or embedding methods?
• RQ4 How does selective annotation (k) and emotion-filter threshold (m) affect performance and efficiency?
• RQ5 Are the model’s explanations (via T^Exp) useful and coherent to users or clinicians?

Key findings

• DORIS achieves higher performance than baselines on AUPRC (0.8134) and AUROC (0.9715).
• DORIS outperforms Mood2Content by 0.0360 in AUPRC on the SWDD dataset.
• Ablation shows removing any component (DC, MC, PH) reduces F1, AUROC, and AUPRC.
• LLM-based direct depression judgments underperform compared to the integrated approach.
• Efficient annotation (annotating 20% of texts) yields substantial AUPRC gains, close to using all texts.
• The system provides explainable outputs (T^Exp) derived from mood course and symptom annotations.

Figure 2: Illustration of DORIS. Through the collaboration of the LLM and the text embedding model, we obtain three key features: depression symptom feature, post history representation, and mood course representation. The classifier uses these three features to make its judgment; the LLM uses annot