[Paper Review] Multimodal Stress Detection Using Facial Landmarks and Biometric Signals

Majid Hosseini, Morteza Bodaghi+3 more|arXiv (Cornell University)|Nov 6, 2023

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes a multimodal stress detection framework fusing facial landmarks (2D-CNN) and biometric signals (1D-CNN) using early and late fusion strategies. It achieves 98.38% accuracy with early fusion, outperforming late fusion (94.39%) and unimodal models, demonstrating that integrated deep learning from diverse physiological and visual cues enhances stress detection reliability and generalizability.

ABSTRACT

The development of various sensing technologies is improving measurements of stress and the well-being of individuals. Although progress has been made with single signal modalities like wearables and facial emotion recognition, integrating multiple modalities provides a more comprehensive understanding of stress, given that stress manifests differently across different people. Multi-modal learning aims to capitalize on the strength of each modality rather than relying on a single signal. Given the complexity of processing and integrating high-dimensional data from limited subjects, more research is needed. Numerous research efforts have been focused on fusing stress and emotion signals at an early stage, e.g., feature-level fusion using basic machine learning methods and 1D-CNN Methods. This paper proposes a multi-modal learning approach for stress detection that integrates facial landmarks and biometric signals. We test this multi-modal integration with various early-fusion and late-fusion techniques to integrate the 1D-CNN model from biometric signals and 2-D CNN using facial landmarks. We evaluate these architectures using a rigorous test of models' generalizability using the leave-one-subject-out mechanism, i.e., all samples related to a single subject are left out to train the model. Our findings show that late-fusion achieved 94.39\% accuracy, and early-fusion surpassed it with a 98.38\% accuracy rate. This research contributes valuable insights into enhancing stress detection through a multi-modal approach. The proposed research offers important knowledge in improving stress detection using a multi-modal approach.

Motivation & Objective

• To improve stress detection accuracy by integrating facial landmark analysis with biometric signals such as heart rate, EDA, and skin temperature.
• To evaluate the effectiveness of early and late fusion techniques in combining heterogeneous modalities (2D-CNN for facial landmarks, 1D-CNN for biometrics) for stress recognition.
• To assess model generalizability using a rigorous leave-one-subject-out cross-validation protocol on a real-world dataset.
• To analyze the impact of feature selection methods and EDA tonic/phasic components on model performance.
• To compare computational costs and trade-offs between model complexity, training time, and inference efficiency.

Proposed method

• Extracted facial landmarks from video data using 2D-CNN to model spatial and temporal dynamics of facial expressions.
• Processed biometric signals (HR, EDA, ST) using 1D-CNN to capture temporal patterns in physiological responses.
• Implemented early fusion by concatenating 1D-CNN and 2D-CNN features before a final fully connected DNN layer.
• Implemented late fusion by independently processing each modality and fusing predictions at the decision layer using element-wise averaging.
• Applied multiple feature selection techniques (Random Forest importance, Recursive Feature Elimination, Variance Threshold) to optimize input features.
• Trained and evaluated models using a leave-one-subject-out cross-validation strategy to ensure robust generalizability.

Figure 1: Illustration of the 68 facial landmarks

Experimental results

Research questions

• RQ1 Does early fusion of 1D-CNN (biometrics) and 2D-CNN (facial landmarks) outperform late fusion in multimodal stress detection?
• RQ2 How do different feature selection methods affect the performance of multimodal stress detection models?
• RQ3 What is the impact of including tonic and phasic components of EDA on model accuracy and reliability?
• RQ4 How do computational costs (training time, parameters, inference time) vary across early and late fusion architectures?
• RQ5 Can multimodal fusion reduce false negative rates in high-stress detection, especially in critical applications?

Key findings

• Early fusion of 1D-CNN (biometrics) and 2D-CNN (facial landmarks) achieved the highest accuracy at 98.38%, significantly outperforming late fusion (94.39%).
• The inclusion of EDA tonic and phasic components improved model performance by 3.57% in accuracy, 9.23% in precision, 10.47% in recall, and 10.71% in F1-score.
• Random Forest-based feature selection yielded 96.94% accuracy, while Variance Threshold reduced performance to 92.76%, highlighting the importance of intelligent feature pruning.
• The 1D-CNN model had the highest number of trainable parameters (23,931) but also the lowest training time (2,590s), indicating a trade-off between complexity and speed.
• Late-fusion models required longer training times (2,220–2,930s) due to independent training of modality-specific networks, despite lower parameter counts.
• The 2D-CNN model on facial landmarks alone achieved strong performance due to its ability to capture spatial and temporal facial dynamics, outperforming unimodal 1D-CNN on biometrics alone.

Figure 2: Phasic and Tonic components of raw EDA signal