[Paper Review] Evaluation of ChatGPT for NLP-based Mental Health Applications

Bishal Lamichhane|arXiv (Cornell University)|Mar 28, 2023

Mental Health via Writing57 citations

TL;DR

The paper evaluates zero-shot ChatGPT (GPT-3.5-turbo) on three mental health text classification tasks (stress, depression, suicidality) using public social media datasets, reporting F1 scores of 0.73, 0.86, and 0.37 respectively, versus simple baselines.

ABSTRACT

Large language models (LLM) have been successful in several natural language understanding tasks and could be relevant for natural language processing (NLP)-based mental health application research. In this work, we report the performance of LLM-based ChatGPT (with gpt-3.5-turbo backend) in three text-based mental health classification tasks: stress detection (2-class classification), depression detection (2-class classification), and suicidality detection (5-class classification). We obtained annotated social media posts for the three classification tasks from public datasets. Then ChatGPT API classified the social media posts with an input prompt for classification. We obtained F1 scores of 0.73, 0.86, and 0.37 for stress detection, depression detection, and suicidality detection, respectively. A baseline model that always predicted the dominant class resulted in F1 scores of 0.35, 0.60, and 0.19. The zero-shot classification accuracy obtained with ChatGPT indicates a potential use of language models for mental health classification tasks.

Motivation & Objective

• Assess zero-shot classification performance of ChatGPT on NLP-based mental health tasks using public social media datasets.
• Compare ChatGPT outputs to baseline dominance models to establish a performance benchmark.
• Analyze confusion patterns and discuss implications for using LLMs as backends in mental health applications.

Proposed method

• Use GPT-3.5-turbo ChatGPT via OpenAI API with a single-class prompt for each post.
• Evaluate three tasks: stress detection (2-class), depression detection (2-class), suicidality detection (5-class).
• Compute F1 scores (weighted in multi-class) and balanced accuracy, and examine confusion matrices for each task.
• Dataset sources: stress detection dataset from Reddit-based posts; depression detection from Reddit and blogs; suicidality detection from a labeled 5-class dataset.
• Compare results to baseline models that always predict the dominant class.

Experimental results

Research questions

• RQ1 Can zero-shot ChatGPT reliably classify social-media text into stress/non-stress, depression/non-depression, and five suicidality-related classes?
• RQ2 How does ChatGPT’s zero-shot performance compare to simple baseline predictors on these mental health tasks?
• RQ3 What do confusion matrices reveal about inter-class confusion, especially in the suicidality five-class setting?],
• RQ4 key_findings:[

Key findings

• Stress detection achieved F1 = 0.73 (baseline 0.35).
• Depression detection achieved F1 = 0.86 (baseline 0.60).
• Suicidality detection achieved F1 = 0.37 (baseline 0.19).
• Balanced accuracy: stress 0.73, depression 0.85, suicidality 0.33.
• Zero-shot ChatGPT shows promising performance relative to baselines, with potential for further improvement via fine-tuning or prompt variation.