[Paper Review] DepreSym: A Depression Symptom Annotated Corpus and the Role of LLMs as Assessors of Psychological Markers

Anxo Pérez, Marcos Fernández-Pichel+2 more|arXiv (Cornell University)|Aug 21, 2023

Mental Health via WritingPsychology3 citations

TL;DR

This paper introduces DepreSym, a dataset of 21,580 sentences annotated for relevance to 21 Beck Depression Inventory-II (BDI-II) symptoms, using a pooling strategy from 37 search systems and expert human assessors. It evaluates GPT-4 and ChatGPT as potential annotators, finding GPT-4 achieves moderate-to-high agreement (kappa = 0.53) and can reduce human labeling effort by ~68% through filtering non-relevant sentences, enabling efficient hybrid annotation workflows for depression symptom detection.

ABSTRACT

Computational methods for depression detection aim to mine traces of depression from online publications posted by Internet users. However, solutions trained on existing collections exhibit limited generalisation and interpretability. To tackle these issues, recent studies have shown that identifying depressive symptoms can lead to more robust models. The eRisk initiative fosters research on this area and has recently proposed a new ranking task focused on developing search methods to find sentences related to depressive symptoms. This search challenge relies on the symptoms specified by the Beck Depression Inventory-II (BDI-II), a questionnaire widely used in clinical practice. Based on the participant systems' results, we present the DepreSym dataset, consisting of 21580 sentences annotated according to their relevance to the 21 BDI-II symptoms. The labelled sentences come from a pool of diverse ranking methods, and the final dataset serves as a valuable resource for advancing the development of models that incorporate depressive markers such as clinical symptoms. Due to the complex nature of this relevance annotation, we designed a robust assessment methodology carried out by three expert assessors (including an expert psychologist). Additionally, we explore here the feasibility of employing recent Large Language Models (ChatGPT and GPT4) as potential assessors in this complex task. We undertake a comprehensive examination of their performance, determine their main limitations and analyze their role as a complement or replacement for human annotators.

Motivation & Objective

• To address the lack of fine-grained, symptom-level annotated data for depression detection in social media.
• To improve model interpretability and generalization by focusing on clinically relevant depressive symptoms rather than general linguistic markers.
• To evaluate the feasibility of using large language models (LLMs) like GPT-4 and ChatGPT as assessors in complex relevance annotation tasks.
• To develop a robust, multi-annotator assessment methodology with expert psychologists to ensure high-quality ground truth.
• To propose a hybrid annotation strategy that leverages LLMs to filter non-relevant sentences, reducing human workload while maintaining quality.

Proposed method

• Constructed a pooled set of candidate sentences from 37 distinct ranking systems participating in the CLEF 2023 eRisk Lab task.
• Employed three expert assessors— including a licensed psychologist—to label sentences as relevant or not based on explicit, symptom-specific content.
• Defined relevance as both topical alignment and explicit mention of an individual’s state related to a BDI-II symptom.
• Applied formal annotation guidelines and conducted inter-rater agreement analysis using Cohen’s kappa and Krippendorff’s alpha.
• Evaluated GPT-4 and ChatGPT as automated assessors by comparing their predictions against human consensus.
• Proposed a hybrid labeling strategy where LLMs pre-filter non-relevant sentences, reducing the burden on human annotators by ~68%.

Experimental results

Research questions

• RQ1 Can large language models like GPT-4 and ChatGPT achieve sufficient agreement with human experts in labeling sentences for depressive symptom relevance?
• RQ2 How does the inter-rater agreement vary among human assessors with different levels of clinical expertise?
• RQ3 To what extent can LLMs reduce human annotation effort without compromising the quality of the final dataset?
• RQ4 How do system rankings based on single-annotator judgments compare to the official consensus-based rankings?
• RQ5 Can a hybrid annotation pipeline using LLMs as filters improve efficiency and scalability in creating clinical symptom-annotated datasets?

Key findings

• GPT-4 achieved a median Cohen’s kappa of 0.53 with human assessors, indicating moderate-to-good agreement, while ChatGPT scored 0.31, suggesting lower reliability.
• The psychologist achieved the highest inter-rater agreement (kappa = 0.54) and the strongest correlation (Kendall’s τ = 0.98) with the official consensus ranking.
• GPT-4 demonstrated high correlation (τ = 0.86, τ_ap = 0.81) with the official system rankings, outperforming individual human assessors and approaching consensus-level reliability.
• The use of GPT-4 as a pre-filter could reduce human labeling workload by approximately 68%, saving an estimated 49 hours per human annotator on the 21,580-sentence dataset.
• Despite strong performance, LLMs still produce significant false positives, indicating they are not yet a full replacement for human experts but are effective as initial screening tools.
• The results support a hybrid annotation strategy where LLMs filter out non-relevant sentences, allowing human experts to focus only on high-potential candidates, improving efficiency and scalability.