[Paper Review] Large-scale Training of Foundation Models for Wearable Biosignals

Salar Abbaspourazad, Oussama Elachqar+4 more|arXiv (Cornell University)|Dec 8, 2023

Mobile Health and mHealth Applications20 citations

TL;DR

The authors pre-train foundation models on large-scale unlabeled PPG and ECG data from the Apple Heart and Movement Study using self-supervised learning with participant-level positives, showing embeddings that encode demographics and health information and differ in utility between PPG and ECG.

ABSTRACT

Tracking biosignals is crucial for monitoring wellness and preempting the development of severe medical conditions. Today, wearable devices can conveniently record various biosignals, creating the opportunity to monitor health status without disruption to one's daily routine. Despite widespread use of wearable devices and existing digital biomarkers, the absence of curated data with annotated medical labels hinders the development of new biomarkers to measure common health conditions. In fact, medical datasets are usually small in comparison to other domains, which is an obstacle for developing neural network models for biosignals. To address this challenge, we have employed self-supervised learning using the unlabeled sensor data collected under informed consent from the large longitudinal Apple Heart and Movement Study (AHMS) to train foundation models for two common biosignals: photoplethysmography (PPG) and electrocardiogram (ECG) recorded on Apple Watch. We curated PPG and ECG datasets from AHMS that include data from ~141K participants spanning ~3 years. Our self-supervised learning framework includes participant level positive pair selection, stochastic augmentation module and a regularized contrastive loss optimized with momentum training, and generalizes well to both PPG and ECG modalities. We show that the pre-trained foundation models readily encode information regarding participants' demographics and health conditions. To the best of our knowledge, this is the first study that builds foundation models using large-scale PPG and ECG data collected via wearable consumer devices $\unicode{x2013}$ prior works have commonly used smaller-size datasets collected in clinical and experimental settings. We believe PPG and ECG foundation models can enhance future wearable devices by reducing the reliance on labeled data and hold the potential to help the users improve their health.

Motivation & Objective

• Address the lack of large, annotated biosignal datasets for robust model development.
• Train foundation models on massive unlabeled PPG and ECG data from wearable devices.
• Investigate what participant-level embeddings reveal about demographics and health conditions.
• Evaluate the impact of self-supervised learning techniques and augmentation strategies on biosignal representations.

Proposed method

• Use contrastive self-supervised learning with InfoNCE and KoLeo regularization to learn embeddings.
• Employ a stochastic, participant-level augmentation module to generate positive pairs from different segments of the same participant.
• Apply momentum training to encoder and projection head to stabilize representations.
• Utilize an EfficientNet-based 1D encoder (256-dim embeddings) and a 128-dim projection head for SSL loss.
• Pre-train on ~20M PPG segments from 141k participants and ~3.75M ECG segments from 106k participants collected via Apple Watch.
• Evaluate linear probes on downstream demographic and health-related targets to assess information encoded in embeddings.

Experimental results

Research questions

• RQ1 Can large-scale self-supervised pre-training on wearable biosignals produce informative embeddings without labels?
• RQ2 Do PPG and ECG embeddings encode participant demographics and health conditions beyond basic heart-rate information?
• RQ3 How do PPG and ECG differ in terms of information content and pre-training dynamics under this framework?
• RQ4 What is the impact of positive-pair selection and augmentation choices on downstream performance?
• RQ5 How do different encoder architectures affect the quality and efficiency of the learned representations?

Key findings

• PPG embeddings nearly always outperform baseline demographic features in predicting age, BMI, and biological sex in linear probing.
• ECG embeddings encode health information but show comparatively lower predictive power for downstream health targets than PPG embeddings.
• Pre-trained embeddings contain information predictive of a broad set of self-reported health conditions and medications.
• Participant-level positive pair selection significantly improves downstream predictive performance over segment-level pairing.
• KoLeo regularization and contrastive/self-supervised framework enhance representation quality; removed KoLeo worsens downstream metrics.
• Different encoder architectures can achieve similar performance, with 1D-EfficientNet offering favorable parameter efficiency.