[Paper Review] See and Read: Detecting Depression Symptoms in Higher Education Students Using Multimodal Social Media Data

Paulo Mann, Aline Paes+1 more|arXiv (Cornell University)|Dec 3, 2019

Mental Health via Writing4 citations

TL;DR

This study proposes a multimodal machine learning approach to detect severe depression symptoms in higher education students using Instagram posts—combining textual captions and visual images. Using both deep learning (ELMo + ResNet-34) and feature engineering models, the best-performing fusion model achieved 0.92 recall and 0.69 precision in identifying students with Beck Depression Inventory scores ≥20, demonstrating the potential of passive, scalable screening via social media data.

ABSTRACT

Mental disorders such as depression and anxiety have been increasing at alarming rates in the worldwide population. Notably, the major depressive disorder has become a common problem among higher education students, aggravated, and maybe even occasioned, by the academic pressures they must face. While the reasons for this alarming situation remain unclear (although widely investigated), the student already facing this problem must receive treatment. To that, it is first necessary to screen the symptoms. The traditional way for that is relying on clinical consultations or answering questionnaires. However, nowadays, the data shared at social media is a ubiquitous source that can be used to detect the depression symptoms even when the student is not able to afford or search for professional care. Previous works have already relied on social media data to detect depression on the general population, usually focusing on either posted images or texts or relying on metadata. In this work, we focus on detecting the severity of the depression symptoms in higher education students, by comparing deep learning to feature engineering models induced from both the pictures and their captions posted on Instagram. The experimental results show that students presenting a BDI score higher or equal than 20 can be detected with 0.92 of recall and 0.69 of precision in the best case, reached by a fusion model. Our findings show the potential of large-scale depression screening, which could shed light upon students at-risk.

Motivation & Objective

• To develop a scalable, passive method for detecting severe depression symptoms in higher education students using social media data.
• To compare the effectiveness of deep learning versus feature engineering models in classifying depression severity from multimodal Instagram content (text and images).
• To evaluate whether fusing textual and visual features improves detection performance over unimodal approaches.
• To explore the feasibility of using publicly available social media data as a proxy for clinical screening, especially for students who may not seek professional help.
• To support early intervention by identifying at-risk students through automated, non-intrusive analysis of user-generated content.

Proposed method

• Collected Instagram posts (images and captions) from higher education students at a large Brazilian university.
• Used the Portuguese translation of the Beck Depression Inventory (BDI) as ground truth, with scores ≥20 indicating severe depression.
• Trained three deep learning models: text-only (ELMo), image-only (ResNet-34), and multimodal fusion (concatenated ELMo + ResNet-34 features).
• Trained three feature engineering models: text-only (TF-IDF + sentiment features), image-only (predefined visual descriptors), and multimodal fusion (concatenated features).
• Evaluated models using precision, recall, and F1-score, with statistical significance assessed via McNemar’s test.
• Employed attention mechanisms and transfer learning to improve interpretability and performance in multimodal learning.

Experimental results

Research questions

• RQ1 Can multimodal analysis of Instagram images and captions effectively detect severe depression symptoms in university students?
• RQ2 How does the performance of deep learning models compare to traditional feature engineering models in detecting depression from multimodal social media data?
• RQ3 Does fusing textual and visual features improve detection accuracy compared to unimodal models?
• RQ4 What is the relative contribution of textual versus visual features in classifying depression severity in this context?
• RQ5 Can passive, automated screening via social media data serve as a viable alternative for early detection in populations with limited access to clinical care?

Key findings

• The best-performing model, a deep multimodal classifier combining ELMo and ResNet-34 features, achieved 0.92 recall and 0.69 precision in identifying students with BDI scores ≥20.
• Text-only models outperformed image-only models in both deep learning and feature engineering approaches, suggesting textual content is more informative for depression detection in this context.
• Fusing visual and textual features via concatenation degraded performance in feature engineering models, indicating a need for better alignment techniques between modalities.
• Deep learning models achieved competitive performance with better generalization potential due to transfer learning, though at the cost of reduced interpretability.
• The study confirms that multimodal social media data can support passive, scalable depression screening in academic populations, especially for those who may not seek clinical help.
• Feature engineering models offer better interpretability, while deep learning models enable easier adaptation to new domains through transfer learning.