[Paper Review] MOGAM: A Multimodal Object-oriented Graph Attention Model for Depression Detection

Junyeop Cha, Seoyun Kim+2 more|arXiv (Cornell University)|Mar 21, 2024

Mental Health via Writing4 citations

TL;DR

This paper proposes MOGAM, a multimodal object-oriented graph attention model that integrates visual objects, metadata, and graph-structured relationships from YouTube vlogs to detect depression and high-risk depression. By leveraging cross-attention to fuse multimodal features and constructing object co-occurrence graphs, MOGAM achieves 87.1% accuracy and 88.8% F1-score on a clinically diagnosed dataset, demonstrating strong performance and scalability.

ABSTRACT

Early detection plays a crucial role in the treatment of depression. Therefore, numerous studies have focused on social media platforms, where individuals express their emotions, aiming to achieve early detection of depression. However, the majority of existing approaches often rely on specific features, leading to limited scalability across different types of social media datasets, such as text, images, or videos. To overcome this limitation, we introduce a Multimodal Object-Oriented Graph Attention Model (MOGAM), which can be applied to diverse types of data, offering a more scalable and versatile solution. Furthermore, to ensure that our model can capture authentic symptoms of depression, we only include vlogs from users with a clinical diagnosis. To leverage the diverse features of vlogs, we adopt a multimodal approach and collect additional metadata such as the title, description, and duration of the vlogs. To effectively aggregate these multimodal features, we employed a cross-attention mechanism. MOGAM achieved an accuracy of 0.871 and an F1-score of 0.888. Moreover, to validate the scalability of MOGAM, we evaluated its performance with a benchmark dataset and achieved comparable results with prior studies (0.61 F1-score). In conclusion, we believe that the proposed model, MOGAM, is an effective solution for detecting depression in social media, offering potential benefits in the early detection and treatment of this mental health condition.

Motivation & Objective

• To develop a scalable, multimodal framework for detecting depression in social media vlogs, particularly from YouTube, using diverse data types.
• To overcome limitations of prior models that rely on specific features (e.g., facial expressions) by introducing an object-oriented graph representation of vlog content.
• To ensure clinical relevance by curating a dataset of vlogs from users with confirmed clinical diagnoses of depression or high-risk potential.
• To evaluate the model’s generalization capability on benchmark datasets, confirming its scalability across different data distributions.
• To provide a robust, interpretable tool for early depression detection using real-world, user-generated vlogs.

Proposed method

• The model constructs an object co-occurrence graph by detecting visual objects (e.g., person, cup, bed) in each vlog using YOLOv5, then computes co-occurrence frequencies between object pairs as graph edges.
• Multimodal features are extracted from three sources: visual objects (via YOLOv5), metadata (title, description, duration) using KoBERT, and visual features (via ResNet) for frames.
• A cross-attention mechanism is applied to dynamically attend to and fuse representations across the three modalities—object graph, visual features, and text metadata.
• The object-oriented graph encoder processes the graph structure using graph attention networks (GATs) to learn contextualized node representations based on neighborhood relationships.
• The final multimodal representation is fed into a classifier head to predict depression or high-risk depression status.
• The model is trained end-to-end with cross-entropy loss and evaluated using standard metrics (accuracy, F1-score).

Experimental results

Research questions

• RQ1 Can a multimodal, object-oriented graph attention model effectively detect depression in YouTube vlogs using diverse data types?
• RQ2 Does the integration of visual objects, metadata, and graph-structured relationships improve detection performance compared to unimodal or non-graph-based approaches?
• RQ3 Can the model generalize to other datasets, including those with different languages or data distributions?
• RQ4 Are specific objects or object co-occurrence patterns significantly associated with depression or high-risk states in vlogs?
• RQ5 Does the use of clinically diagnosed vlogs enhance the model’s reliability and clinical relevance?

Key findings

• MOGAM achieved an accuracy of 87.1% and an F1-score of 88.8% on the primary dataset of vlogs from individuals with clinical depression or high-risk potential.
• The model demonstrated strong scalability, achieving a 0.61 F1-score on a benchmark dataset, comparable to prior state-of-the-art methods.
• Objects such as knife, fork, cake, handbag, and sandwich showed statistically significant associations with depression states, with p-values indicating strong discriminative power.
• The frequency of object appearances was higher in the daily group compared to the depression and high-risk potential groups, suggesting altered object usage patterns in clinical states.
• The top five objects with the lowest p-values (knife, fork, cake, handbag, sandwich) had distinct mean appearance counts across groups, indicating potential diagnostic value.
• Post-hoc analysis confirmed that food-related objects were significantly more prevalent in the depression and high-risk groups, suggesting their relevance in symptom expression.