[Paper Review] A Quantitative and Qualitative Analysis of Suicide Ideation Detection using Deep Learning

Siqu Long, Rina Carines Cabral+2 more|arXiv (Cornell University)|Jun 17, 2022

Mental Health via Writing4 citations

TL;DR

This study replicates and evaluates state-of-the-art deep learning models—RNN, CNN, and attention-based networks—on multiple suicide ideation detection datasets using diverse input embeddings and pretrained language models. It finds that model performance heavily depends on dataset quality, with pretrained models like RoBERTa outperforming traditional architectures, especially on larger, contextually rich datasets, and highlights the critical role of data quality and contextual understanding in suicide risk detection from social media text.

ABSTRACT

For preventing youth suicide, social media platforms have received much attention from researchers. A few researches apply machine learning, or deep learning-based text classification approaches to classify social media posts containing suicidality risk. This paper replicated competitive social media-based suicidality detection/prediction models. We evaluated the feasibility of detecting suicidal ideation using multiple datasets and different state-of-the-art deep learning models, RNN-, CNN-, and Attention-based models. Using two suicidality evaluation datasets, we evaluated 28 combinations of 7 input embeddings with 4 commonly used deep learning models and 5 pretrained language models in quantitative and qualitative ways. Our replication study confirms that deep learning works well for social media-based suicidality detection in general, but it highly depends on the dataset's quality.

Motivation & Objective

• To address the lack of reproducible and generalizable suicide ideation detection models in social media by replicating competitive existing approaches.
• To evaluate the performance of diverse deep learning models—RNN, CNN, and attention-based networks—across multiple publicly available suicidality datasets.
• To isolate the impact of textual features by comparing results using only text embeddings against other multimodal or feature-integrated approaches.
• To promote research transparency by releasing full code and detailed hyperparameter configurations to enable replication and further development.
• To provide a comprehensive comparative analysis of embedding types, deep learning architectures, and pretrained language models on suicidality detection tasks.

Proposed method

• The study uses two distinct suicidality detection datasets: the Suicide-related Tweets Dataset (based on keyword-matching and post-level annotation) and the UMD Reddit Suicidality Dataset (based on user-level risk labels, not post-level).
• It evaluates 28 combinations of 7 input embeddings (e.g., word, char, doc, and hybrid) with 4 traditional deep learning models (Bi-LSTM, CNN, GRU, and Attention-based models).
• It further evaluates 5 pretrained language models (including BERT and RoBERTa) across both datasets to compare their performance against baseline models and traditional architectures.
• The evaluation uses standard classification metrics (accuracy, F1, AUC) and includes qualitative case studies with probability heatmaps from the softmax layer to interpret model predictions.
• A 5-fold cross-validation and 80-20 train-test split are used, with proportional sampling to preserve label distribution in the UMD Reddit dataset.
• The study emphasizes reproducibility by releasing full code and detailed hyperparameter settings for all experiments.

Experimental results

Research questions

• RQ1 How do different deep learning architectures (RNN, CNN, attention-based) perform on suicide ideation detection across multiple datasets?
• RQ2 What is the impact of various input embeddings (e.g., word, char, doc, and hybrid) on model performance in suicidality detection?
• RQ3 How do pretrained language models like BERT and RoBERTa compare to traditional deep learning models and baseline classifiers in detecting suicidal ideation?
• RQ4 To what extent does dataset quality and annotation strategy (post-level vs. user-level) influence model performance and generalizability?
• RQ5 Can qualitative analysis of prediction probabilities reveal model reasoning patterns, especially in ambiguous or context-dependent cases?

Key findings

• On the Suicide-related Tweets Dataset, attention-based models with combined char+doc embeddings achieved the highest accuracy, while traditional models showed consistent performance patterns across embedding types.
• On the UMD Reddit Suicidality Dataset, RoBERTa outperformed all other models with an accuracy in the 45%-60% range, significantly surpassing baseline models.
• The performance of traditional deep learning models (e.g., Bi-LSTM, CNN) was highly dependent on embedding type, with doc embeddings yielding the lowest accuracy and hybrid embeddings the highest.
• Baseline models (Rotation Forest and SVM) performed poorly on the Reddit dataset, especially when suicide-related keywords appeared in non-clinical contexts, such as 'dead' in 'keg of Franziskaner is dead'.
• RoBERTa correctly classified a post as 'no risk' despite containing the word 'dead', demonstrating its ability to leverage contextual understanding, unlike simpler models.
• The study confirms that deep learning models are effective for suicide ideation detection only when sufficient, high-quality data is available, particularly in the specific class of interest.