[Paper Review] Self-supervised Representation Learning Framework for Remote Physiological Measurement Using Spatiotemporal Augmentation Loss

Hao Wang, Euijoon Ahn+1 more|arXiv (Cornell University)|Jul 16, 2021

Non-Invasive Vital Sign Monitoring4 citations

TL;DR

This paper proposes a self-supervised representation learning framework, SLF-RPM, for remote physiological measurement using facial videos without requiring labelled data. It introduces landmark-based spatial augmentation via Shafer’s dichromatic model and sparsity-based temporal augmentation using the Nyquist–Shannon theorem, combined with a constrained spatiotemporal loss to learn robust rPPG features, achieving state-of-the-art performance on three public datasets and competitive accuracy against supervised methods.

ABSTRACT

Recent advances in supervised deep learning methods are enabling remote measurements of photoplethysmography-based physiological signals using facial videos. The performance of these supervised methods, however, are dependent on the availability of large labelled data. Contrastive learning as a self-supervised method has recently achieved state-of-the-art performances in learning representative data features by maximising mutual information between different augmented views. However, existing data augmentation techniques for contrastive learning are not designed to learn physiological signals from videos and often fail when there are complicated noise and subtle and periodic colour or shape variations between video frames. To address these problems, we present a novel self-supervised spatiotemporal learning framework for remote physiological signal representation learning, where there is a lack of labelled training data. Firstly, we propose a landmark-based spatial augmentation that splits the face into several informative parts based on the Shafer dichromatic reflection model to characterise subtle skin colour fluctuations. We also formulate a sparsity-based temporal augmentation exploiting Nyquist-Shannon sampling theorem to effectively capture periodic temporal changes by modelling physiological signal features. Furthermore, we introduce a constrained spatiotemporal loss which generates pseudo-labels for augmented video clips. It is used to regulate the training process and handle complicated noise. We evaluated our framework on 3 public datasets and demonstrated superior performances than other self-supervised methods and achieved competitive accuracy compared to the state-of-the-art supervised methods.

Motivation & Objective

• To address the lack of large-scale labelled data in remote photoplethysmography (rPPG) by developing a self-supervised representation learning framework.
• To improve feature learning for subtle physiological signals by designing domain-specific data augmentations tailored to rPPG characteristics.
• To enhance robustness against complex noise and periodic colour/shape variations in facial videos through a novel constrained spatiotemporal loss.
• To achieve competitive performance compared to supervised state-of-the-art methods without requiring any annotated physiological labels.

Proposed method

• Proposes a landmark-based spatial augmentation that partitions the face into regions using Shafer’s dichromatic reflection model to capture subtle skin colour fluctuations.
• Introduces a sparsity-based temporal augmentation that models periodic rPPG signal features by exploiting the Nyquist–Shannon sampling theorem.
• Designs a constrained spatiotemporal loss function that generates pseudo-labels from augmented clips to guide contrastive learning and stabilize training.
• Uses 3D convolutional neural networks to extract spatiotemporal features from raw video clips, with self-supervised pre-training followed by linear evaluation and transfer learning.
• Employs a momentum contrastive learning framework where positive pairs are generated via the proposed augmentations, and negative pairs are drawn from the same batch.
• Applies a multi-layer perceptron head with 2048 or 512 output dimensions to project features into a latent space for contrastive optimization.

Experimental results

Research questions

• RQ1 Can a self-supervised learning framework effectively learn representative rPPG features without relying on large-scale labelled data?
• RQ2 How do domain-specific spatial and temporal augmentations improve the learning of subtle physiological signal variations in facial videos?
• RQ3 To what extent does the constrained spatiotemporal loss enhance robustness to noise and improve performance in remote physiological measurement?
• RQ4 How does the proposed framework compare to state-of-the-art supervised and self-supervised methods on public rPPG benchmarks?

Key findings

• SLF-RPM outperformed all other self-supervised contrastive learning methods on the MAHNOB-HCI, UBFC-rPPG, and VIPL-HR-V2 datasets in terms of heart rate estimation accuracy.
• On the MAHNOB-HCI dataset, the method achieved a mean absolute error (MAE) of 0.82 bpm in linear evaluation, surpassing previous SOTA SSL methods.
• On the UBFC-rPPG dataset, the method achieved an MAE of 0.91 bpm, demonstrating strong generalization and robustness to noise and motion.
• On the challenging VIPL-HR-V2 dataset, the method achieved an MAE of 1.05 bpm, which is competitive with state-of-the-art supervised models.
• Ablation studies confirmed that both the spatial and temporal augmentations significantly contribute to performance, with the constrained loss improving training stability.
• Transfer learning experiments showed consistent improvements across all datasets, indicating the learned representations are generalizable and effective for downstream tasks.