[Paper Review] Enhancing Performance and User Engagement in Everyday Stress Monitoring: A Context-Aware Active Reinforcement Learning Approach

Seyed Amir Hossein Aqajari, Ziyu Wang+7 more|arXiv (Cornell University)|Jul 11, 2024

Mental Health Research Topics6 citations

TL;DR

The paper proposes a context-aware deep Q-learning-based active reinforcement learning framework to optimally time Ecological Momentary Assessments (EMAs) for stress labeling using PPG from wearables and contextual smartphone data, improving efficiency, accuracy, and personalization in online deployment.

ABSTRACT

In today's fast-paced world, accurately monitoring stress levels is crucial. Sensor-based stress monitoring systems often need large datasets for training effective models. However, individual-specific models are necessary for personalized and interactive scenarios. Traditional methods like Ecological Momentary Assessments (EMAs) assess stress but struggle with efficient data collection without burdening users. The challenge is to timely send EMAs, especially during stress, balancing monitoring efficiency and user convenience. This paper introduces a novel context-aware active reinforcement learning (RL) algorithm for enhanced stress detection using Photoplethysmography (PPG) data from smartwatches and contextual data from smartphones. Our approach dynamically selects optimal times for deploying EMAs, utilizing the user's immediate context to maximize label accuracy and minimize intrusiveness. Initially, the study was executed in an offline environment to refine the label collection process, aiming to increase accuracy while reducing user burden. Later, we integrated a real-time label collection mechanism, transitioning to an online methodology. This shift resulted in an 11% improvement in stress detection efficiency. Incorporating contextual data improved model accuracy by 4%. Personalization studies indicated a 10% enhancement in AUC-ROC scores, demonstrating better stress level differentiation. This research marks a significant move towards personalized, context-driven real-time stress monitoring methods.

Motivation & Objective

• Motivate accurate, personalized daily stress monitoring using biosignals (PPG) from wearables and contextual smartphone data.
• Reduce user burden by optimizing when to trigger EMAs during daily activities.
• Extend from offline to online learning to improve labeling efficiency and model performance.
• Evaluate personalization effects on stress detection performance across individuals.

Proposed method

• Develop a three-layer sensor-edge-cloud system (ZotCare) for data collection, processing, and EMA triggering.
• Use a context-aware Deep Q-Network (DQN) to decide whether to query for stress labels, incorporating uncertainty, time-aware responsiveness, and time since last query into the state.
• Define a reward structure that blends classifier uncertainty, user responsiveness, and timing considerations to guide EMA queries.
• Preprocess PPG signals with filtering, smoothing, normalization, and extract 12 features via HeartPy for model input.
• Compare offline context-aware RL with random and traditional active learning, then implement online RL with real-time querying and personalization.

Figure 1. System Architecture - Offline Study

Experimental results

Research questions

• RQ1 Can context-aware active reinforcement learning reduce unnecessary EMAs while maintaining or improving stress detection performance?
• RQ2 Does online RL improve labeling efficiency and stress detection accuracy compared with offline RL?
• RQ3 How does personalization (subject-specific data) affect stress detection performance in RL-based labeling?
• RQ4 What is the impact of incorporating contextual features and temporal data correlations on EMA scheduling and model performance?

Key findings

• Online RL yielded an 11% improvement in stress detection efficiency over offline methods.
• Incorporating contextual data improved model accuracy by 4%.
• Personalization studies showed a 10% enhancement in AUC-ROC scores.
• Context-aware RL reduced EMA queries by up to 88% compared with random labeling in personalization scenarios.
• The offline study collected 132,598 filtered samples from 54 participants across two IRB-approved phases (June 2020–June 2021 and March 2022–May 2023).
• Two-stage IRB-approved protocol and public data release plan for datasets.

Figure 2. Number of queries needed to reach a certain performance level during personalization.