[Paper Review] Personalized Prediction of Recurrent Stress Events Using Self-Supervised Learning on Multimodal Time-Series Data

Tanvir Islam, Peter Washington|arXiv (Cornell University)|Jul 7, 2023

Mental Health Research Topics8 citations

TL;DR

The paper presents a personalized, self-supervised learning framework that pre-trains on individual multimodal biosignals to predict stress with minimal annotations, evaluated on the WESAD dataset.

ABSTRACT

Chronic stress can significantly affect physical and mental health. The advent of wearable technology allows for the tracking of physiological signals, potentially leading to innovative stress prediction and intervention methods. However, challenges such as label scarcity and data heterogeneity render stress prediction difficult in practice. To counter these issues, we have developed a multimodal personalized stress prediction system using wearable biosignal data. We employ self-supervised learning (SSL) to pre-train the models on each subject's data, allowing the models to learn the baseline dynamics of the participant's biosignals prior to fine-tuning the stress prediction task. We test our model on the Wearable Stress and Affect Detection (WESAD) dataset, demonstrating that our SSL models outperform non-SSL models while utilizing less than 5% of the annotations. These results suggest that our approach can personalize stress prediction to each user with minimal annotations. This paradigm has the potential to enable personalized prediction of a variety of recurring health events using complex multimodal data streams.

Motivation & Objective

• Address label scarcity and data heterogeneity in stress prediction from multimodal biosignals.
• Develop a personalized model trained per user to capture individual baseline dynamics.
• Leverage self-supervised pre-training to learn robust representations from unlabeled data.
• Fuse multiple biosignals (EDA, ECG, EMG, TEMP, RESP, ACC) for improved stress prediction.
• Demonstrate reduced annotation requirements while maintaining or improving performance.

Proposed method

• Segment raw multimodal biosignal data into overlapping windows for model input.
• Use a subject-specific self-supervised pre-training task to learn baseline representations from unlabeled data.
• Fine-tune the pre-trained model on the downstream stress prediction task using labeled data.
• Fuse six modalities through late-stage fusion to produce a multimodal representation for stress prediction.
• Train a two-stage architecture with frozen pre-trained layers and task-specific dense layers.
• Evaluate using RMSE and compare SSL pre-training against purely supervised training.

Figure 1: Personalized stress prediction framework involves segmenting the raw signal into overlapping data points and utilizing a self-supervised pre-training forecasting model (right model), which is then fine-tuned for stress prediction. We compare this model to a supervised learning model (left

Experimental results

Research questions

• RQ1 Can per-user self-supervised pre-training improve stress prediction over non-SSL approaches with limited annotations?
• RQ2 How does multimodal personalization influence performance compared to single-modality or non-personalized models?
• RQ3 Does SSL reduce the required number of labeled examples to achieve accurate stress predictions?

Key findings

• SSL pre-training yields significantly lower RMSE than purely supervised models across all subjects and annotation types.
• SSL models outperform non-SSL models even when trained with as few as 5 randomly sampled data points.
• Personalized, multimodal SSL reduces annotation burden while preserving predictive accuracy.
• Fused multimodal representations from six biosignals improve stress prediction performance.
• The approach demonstrates robust improvements in a controlled dataset (WESAD) using subject-specific baselines.

Figure 2: RMSE between a model fine-tuned after SSL and a model trained purely with supervised training for all subjects and all annotation types.