[Paper Review] A multimodal stress detection dataset with facial expressions and physiological signals

Majid Hosseini, Fahad Sohrab+6 more|arXiv (Cornell University)|Aug 29, 2022

Emotion and Mood Recognition7 citations

TL;DR

This paper introduces EmpathicSchool, a novel multimodal dataset capturing facial expressions and physiological signals—such as heart rate, electrodermal activity, and skin temperature—under controlled stress conditions from 20 participants across 26 hours. The dataset enables multimodal stress detection by correlating facial emotion cues with biometric signals, with statistical validation showing significant differences in heart rate and specific emotions (e.g., angry, surprised, sad) across stress levels (p < 0.05).

ABSTRACT

Affective computing has garnered the attention and interest of researchers in recent years, as there is a need for AI systems to better understand and react to human emotions. However, analyzing human emotions, such as mood or stress, is quite complex. While various stress studies use facial expressions and wearables, most existing datasets rely on processing data from a single modality. This paper presents EmpathicSchool, a novel dataset that captures facial expressions and the associated physiological signals, such as heart rate, electrodermal activity, and skin temperature, under different stress levels. The data was collected from 30 participants during different sessions for about ninety minutes each (for a total of 40 hours). The data includes seven different signal types, including both computer vision and physiological features that can be used to detect stress. In addition, various experiments were conducted to validate the signal quality.

Motivation & Objective

• To address the lack of multimodal datasets that jointly analyze facial expressions and physiological signals for stress detection.
• To create a high-fidelity, real-time dataset capturing both emotional expressions and biometric responses under varying stress levels.
• To validate signal quality and establish statistical correlations between specific emotions and stress-related physiological changes.
• To support the development of empathic systems in education, healthcare, and workplace wellness by providing labeled, multimodal data.
• To enable research in affective computing by offering a publicly available dataset with both video and sensor data for stress and emotion analysis.

Proposed method

• Collected data from 20 participants across multiple sessions over 26 hours using synchronized video recording and wearable sensors.
• Captured nine distinct signal types, including facial features (via InceptionV3 and dlib 68-point landmarks) and physiological signals (ECG, EDA, skin temperature, etc.).
• Used a controlled experimental setup with three stress-level sessions: low, moderate, and high stress, with labeled emotional states.
• Applied one-way ANOVA to test the significance of differences in physiological signals across stress levels and emotional categories.
• Leveraged pre-trained models (InceptionV3 on FER-2013) and dlib for facial feature extraction to ensure accurate emotion recognition from video.
• Provided open-source code for facial expression analysis and physiological signal visualization, enabling reproducibility and further analysis.

Experimental results

Research questions

• RQ1 Can facial expressions and physiological signals be reliably synchronized and labeled in a multimodal dataset under varying stress conditions?
• RQ2 Which physiological signals (e.g., heart rate, EDA, skin temperature) show statistically significant differences across distinct stress levels?
• RQ3 How do specific facial emotions (e.g., anger, surprise, sadness) correlate with measurable changes in stress-related biometrics?
• RQ4 To what extent do emotional states and physiological responses co-vary during acute stress exposure in a controlled environment?
• RQ5 Can this dataset support the development of real-time, multimodal stress detection systems in applications like online education or ICU monitoring?

Key findings

• Heart rate showed a statistically significant difference across stress levels (F = 4.346, p = 0.037), supporting its use as a reliable stress indicator.
• Emotions such as angry (p = 0.004), surprised (p = 0.022), and sad (p = 0.030) were significantly correlated with stress, indicating strong multimodal signal relationships.
• Electrodermal activity (EDA), skin temperature, and neutral/happy/scared/disgust states showed no significant differences across stress levels (p > 0.05), suggesting limited discriminative power in this dataset.
• A sudden drop in skin temperature was observed during early stress onset, though it later increased, indicating a complex temporal relationship with stress.
• The dataset demonstrated high signal quality and consistency, validated through repeated measurements and statistical analysis across participants.
• The combination of facial expression analysis (using InceptionV3 and dlib) and physiological signal processing enabled robust, real-time multimodal feature extraction for stress modeling.