[Paper Review] Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media

Loukas Ilias, Spiros Mouzakitis+1 more|arXiv (Cornell University)|May 26, 2023

Mental Health via Writing4 citations

TL;DR

This paper proposes a novel method to improve stress and depression detection in social media by injecting linguistic features—such as LIWC, NRC, LDA topics (GOSS), and Top2Vec—into BERT and MentalBERT via a Multimodal Adaptation Gate, while applying label smoothing to enhance model calibration. Results show significant performance gains, especially with LIWC and GOSS features, and label smoothing improves both accuracy and calibration across three datasets.

ABSTRACT

In today's fast-paced world, the rates of stress and depression present a surge. Social media provide assistance for the early detection of mental health conditions. Existing methods mainly introduce feature extraction approaches and train shallow machine learning classifiers. Other researches use deep neural networks or transformers. Despite the fact that transformer-based models achieve noticeable improvements, they cannot often capture rich factual knowledge. Although there have been proposed a number of studies aiming to enhance the pretrained transformer-based models with extra information or additional modalities, no prior work has exploited these modifications for detecting stress and depression through social media. In addition, although the reliability of a machine learning model's confidence in its predictions is critical for high-risk applications, there is no prior work taken into consideration the model calibration. To resolve the above issues, we present the first study in the task of depression and stress detection in social media, which injects extra linguistic information in transformer-based models, namely BERT and MentalBERT. Specifically, the proposed approach employs a Multimodal Adaptation Gate for creating the combined embeddings, which are given as input to a BERT (or MentalBERT) model. For taking into account the model calibration, we apply label smoothing. We test our proposed approaches in three publicly available datasets and demonstrate that the integration of linguistic features into transformer-based models presents a surge in the performance. Also, the usage of label smoothing contributes to both the improvement of the model's performance and the calibration of the model. We finally perform a linguistic analysis of the posts and show differences in language between stressful and non-stressful texts, as well as depressive and non-depressive posts.

Motivation & Objective

• To address the lack of reliable confidence estimation in mental health detection models by introducing model calibration via label smoothing.
• To enhance transformer-based models (BERT, MentalBERT) with external linguistic features for improved detection of stress and depression in social media.
• To investigate the linguistic differences between stressful/depressive and non-stressful/non-depressive posts through feature analysis.
• To explore the use of Global Outlier Standard Score (GOSS) for capturing topic interest in depression/stress detection.
• To develop a multimodal fusion mechanism using a Multimodal Adaptation Gate that dynamically weights transformer and linguistic representations.

Proposed method

• Extract multiple linguistic features: LIWC, NRC Sentiment Lexicon, Top2Vec embeddings, and LDA-based GOSS features for each social media post.
• Project all linguistic features into the same embedding space as BERT/MentalBERT outputs via learned linear projections.
• Concatenate the projected linguistic features with the [CLS] token representation from BERT/MentalBERT to form a unified input.
• Apply a Multimodal Adaptation Gate with learnable attention weights to dynamically control the contribution of each modality (transformer and linguistic features).
• Fine-tune the combined model using label smoothing to improve both classification performance and prediction calibration.
• Evaluate model calibration using Expected Calibration Error (ECE) and Adaptive Calibration Error (Adaptive ECE).

Experimental results

Research questions

• RQ1 Can injecting linguistic features into BERT and MentalBERT improve the detection of stress and depression in social media texts?
• RQ2 Does label smoothing improve both the classification performance and the calibration of transformer-based models in mental health detection?
• RQ3 Which linguistic feature types (e.g., LIWC, NRC, GOSS) contribute most significantly to improved model performance?
• RQ4 How do the linguistic patterns of depressive and stressful posts differ from non-depressive and non-stressful posts?
• RQ5 To what extent does the Multimodal Adaptation Gate enhance feature fusion compared to simple concatenation?

Key findings

• The integration of LIWC features into MentalBERT with label smoothing improved F1-score by 3.36% compared to standard MentalBERT and by 0.63% compared to M-MentalBERT (LIWC) without label smoothing.
• Using top2vec features with BERT and label smoothing achieved the highest F1-score and accuracy on the Depression_Mixed dataset.
• On the Dreaddit dataset, LIWC feature integration with label smoothing yielded the highest F1-score and accuracy across all configurations.
• For the Depression_Severity dataset, NRC features with BERT and GOSS features with MentalBERT achieved the highest weighted F1-scores.
• Linguistic analysis revealed strong correlations between depressive and stressful posts and specific LIWC categories, such as negative emotion and cognitive processes.
• Label smoothing significantly improved model calibration, as measured by reduced Expected Calibration Error and Adaptive Calibration Error across all datasets.