[Paper Review] Data set creation and empirical analysis for detecting signs of depression from social media postings

S Kayalvizhi, D Thenmozhi|arXiv (Cornell University)|Feb 7, 2022

Mental Health via Writing18 citations

TL;DR

This paper introduces a novel, publicly available dataset for detecting three levels of depression—'not depressed', 'moderately depressed', and 'severely depressed'—from social media text using Reddit posts. It employs traditional machine learning models with data augmentation via SMOTE, achieving an F1-score and accuracy of 0.877 using Word2Vec embeddings and a Random Forest classifier, significantly improving performance on imbalanced data.

ABSTRACT

Depression is a common mental illness that has to be detected and treated at an early stage to avoid serious consequences. There are many methods and modalities for detecting depression that involves physical examination of the individual. However, diagnosing mental health using their social media data is more effective as it avoids such physical examinations. Also, people express their emotions well in social media, it is desirable to diagnose their mental health using social media data. Though there are many existing systems that detects mental illness of a person by analysing their social media data, detecting the level of depression is also important for further treatment. Thus, in this research, we developed a gold standard data set that detects the levels of depression as `not depressed', `moderately depressed' and `severely depressed' from the social media postings. Traditional learning algorithms were employed on this data set and an empirical analysis was presented in this paper. Data augmentation technique was applied to overcome the data imbalance. Among the several variations that are implemented, the model with Word2Vec vectorizer and Random Forest classifier on augmented data outperforms the other variations with a score of 0.877 for both accuracy and F1 measure.

Motivation & Objective

• To address the research gap in detecting not just depression, but also its severity levels, from social media text.
• To create a gold-standard, publicly available dataset annotated at three depression severity levels: not depressed, moderately depressed, and severely depressed.
• To empirically evaluate traditional machine learning models on this dataset and assess the impact of data augmentation on model performance.
• To provide a benchmark for future research in computational mental health using social media data.

Proposed method

• Collected social media text from Reddit subreddits focused on mental health, such as r/depression and r/Anxiety.
• Annotated each post into one of three classes: 'not depressed', 'moderately depressed', or 'severely depressed' using domain experts.
• Extracted text features using three embedding techniques: TF-IDF, GloVe, and Word2Vec.
• Applied Synthetic Minority Oversampling Technique (SMOTE) to balance the imbalanced dataset, particularly addressing the overrepresentation of the 'moderately depressed' class.
• Trained and evaluated multiple traditional classifiers, including Random Forest, SVM, KNN, Logistic Regression, and MLP, on both raw and augmented data.
• Compared model performance using metrics such as F1-score and accuracy, with a focus on the impact of data augmentation.

Experimental results

Research questions

• RQ1 Can a multi-class dataset for depression severity levels be effectively constructed from social media text using expert-annotated Reddit posts?
• RQ2 How do traditional machine learning models perform in classifying depression severity levels when trained on this new dataset?
• RQ3 To what extent does data augmentation via SMOTE improve model performance on this imbalanced depression severity dataset?
• RQ4 Which text representation method (TF-IDF, GloVe, Word2Vec) yields the best performance when combined with a given classifier on this dataset?
• RQ5 What is the optimal model configuration (embedding + classifier) for detecting levels of depression from social media text?

Key findings

• The model using Word2Vec embeddings and a Random Forest classifier achieved the highest performance on the augmented dataset, with both F1-score and accuracy reaching 0.877.
• Data augmentation using SMOTE improved the F1-score by 23% and accuracy by 12% compared to baseline models on raw data.
• Among the classifiers tested, Random Forest consistently outperformed others, especially after data augmentation, achieving the best F1-score of 0.877.
• The 'moderately depressed' class was significantly more prevalent than the other two classes, indicating a need for data balancing techniques such as SMOTE.
• The model with Word2Vec and Random Forest showed robustness and generalization, outperforming models using TF-IDF and GloVe on the same dataset.
• The dataset is publicly available on GitHub, enabling reproducibility and future benchmarking in computational mental health research.