[Paper Review] Depression Status Estimation by Deep Learning based Hybrid Multi-Modal Fusion Model

Hrithwik Shalu, Pandurangan Harikrishnan+7 more|arXiv (Cornell University)|Nov 30, 2020

Mental Health via Writing5 references4 citations

TL;DR

This paper proposes a hybrid deep learning model that fuses audio, video, and text modalities using one-shot learning and supervised fine-tuning to improve depression status estimation. The model achieves 96.3% accuracy and 0.9682 AUC on a real-world dataset, demonstrating robust performance in detecting mild depression through user-adaptive, cloud-deployed smartphone application integration.

ABSTRACT

Preliminary detection of mild depression could immensely help in effective treatment of the common mental health disorder. Due to the lack of proper awareness and the ample mix of stigmas and misconceptions present within the society, mental health status estimation has become a truly difficult task. Due to the immense variations in character level traits from person to person, traditional deep learning methods fail to generalize in a real world setting. In our study we aim to create a human allied AI workflow which could efficiently adapt to specific users and effectively perform in real world scenarios. We propose a Hybrid deep learning approach that combines the essence of one shot learning, classical supervised deep learning methods and human allied interactions for adaptation. In order to capture maximum information and make efficient diagnosis video, audio, and text modalities are utilized. Our Hybrid Fusion model achieved a high accuracy of 96.3% on the Dataset; and attained an AUC of 0.9682 which proves its robustness in discriminating classes in complex real-world scenarios making sure that no cases of mild depression are missed during diagnosis. The proposed method is deployed in a cloud-based smartphone application for robust testing. With user-specific adaptations and state of the art methodologies, we present a state-of-the-art model with user friendly experience.

Motivation & Objective

• To address the challenge of undetected mild depression due to stigma, low awareness, and individual variability in symptoms.
• To develop a user-adaptive AI system that generalizes well across diverse individuals in real-world settings.
• To improve early detection of depression by leveraging multimodal data (audio, video, text) with hybrid learning strategies.
• To create a clinically useful, deployable tool for preliminary depression screening accessible via smartphone.

Proposed method

• The model uses a hybrid fusion framework combining one-shot learning for user-specific adaptation with supervised deep learning on multimodal inputs.
• Audio features are extracted using prosodic and spectral analysis, capturing speech rate, pauses, and voice quality.
• Video features are derived from facial expressions and micro-expressions using 3D CNNs to detect emotional cues.
• Text transcripts are processed using NLP techniques to identify linguistic markers such as first-person pronouns and syntactic simplicity.
• Cosine similarity is used to compare query responses with expert-labeled standard answers to determine depression severity.
• The system is deployed in a cloud-based smartphone application enabling real-time, scalable, and user-adaptive screening.

Experimental results

Research questions

• RQ1 Can a hybrid deep learning model effectively fuse audio, video, and text modalities to detect mild depression with high accuracy in real-world settings?
• RQ2 How well can one-shot learning techniques enable user-specific adaptation in depression detection without requiring large labeled datasets per individual?
• RQ3 To what extent does multimodal fusion improve discrimination between depression severity levels compared to single-modality approaches?
• RQ4 Can the model generalize across diverse individuals with varying personality and behavioral traits in real-world clinical scenarios?
• RQ5 How effective is the system in identifying subtle or early signs of depression that are often missed by traditional clinical methods?

Key findings

• The proposed hybrid fusion model achieved a test accuracy of 96.3% with a 95% confidence interval of ±1.478.
• The model demonstrated an AUC of 0.9682, indicating strong discriminative power between depression classes in complex real-world data.
• Cosine similarity indices ranged from 0.128 to 0.883, confirming the model’s ability to distinguish between mild, moderate, and severe depression levels.
• The system successfully detected subtle linguistic, prosodic, and facial indicators of depression across diverse individuals.
• The cloud-based smartphone deployment enabled scalable, real-time, and user-adaptive screening, enhancing accessibility and clinical utility.
• Expert validation confirmed that the AI-aided system can assist clinicians in prioritizing cases and improving triage efficiency.