[Paper Review] Contrast-Phys+: Unsupervised and Weakly-supervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast

Zhaodong Sun, Xiaobai Li|arXiv (Cornell University)|Sep 13, 2023

Non-Invasive Vital Sign Monitoring80 references5 citations

TL;DR

Contrast-Phys+ is a novel unsupervised and weakly-supervised video-based remote photoplethysmography (rPPG) method that leverages spatiotemporal contrastive learning with rPPG-specific priors—spatial similarity, temporal consistency, cross-video dissimilarity, and heart rate range constraints. It achieves state-of-the-art performance even with no ground truth labels or with misaligned/missing labels, outperforming supervised SOTA methods in accuracy, noise robustness, and computational efficiency.

ABSTRACT

Video-based remote physiological measurement utilizes facial videos to measure the blood volume change signal, which is also called remote photoplethysmography (rPPG). Supervised methods for rPPG measurements have been shown to achieve good performance. However, the drawback of these methods is that they require facial videos with ground truth (GT) physiological signals, which are often costly and difficult to obtain. In this paper, we propose Contrast-Phys+, a method that can be trained in both unsupervised and weakly-supervised settings. We employ a 3DCNN model to generate multiple spatiotemporal rPPG signals and incorporate prior knowledge of rPPG into a contrastive loss function. We further incorporate the GT signals into contrastive learning to adapt to partial or misaligned labels. The contrastive loss encourages rPPG/GT signals from the same video to be grouped together, while pushing those from different videos apart. We evaluate our methods on five publicly available datasets that include both RGB and Near-infrared videos. Contrast-Phys+ outperforms the state-of-the-art supervised methods, even when using partially available or misaligned GT signals, or no labels at all. Additionally, we highlight the advantages of our methods in terms of computational efficiency, noise robustness, and generalization. Our code is available at https://github.com/zhaodongsun/contrast-phys.

Motivation & Objective

• To address the high cost and difficulty of acquiring ground truth (GT) physiological signals for supervised rPPG training.
• To develop a unified framework that works effectively under unsupervised, weakly-supervised, and semi-supervised settings.
• To improve robustness to noise, head motion, and label misalignment without requiring fully labeled data.
• To enhance generalization and computational efficiency in real-world rPPG applications.

Proposed method

• Uses a 3D CNN to extract multiple spatiotemporal rPPG signals from facial video clips.
• Designs a contrastive loss function that groups rPPG signals from the same video while pushing apart signals from different videos.
• Incorporates rPPG priors—spatial similarity, temporal consistency, cross-video dissimilarity, and HR range (40–250 bpm)—into the contrastive objective.
• Adapts to weak supervision by integrating partial or misaligned GT signals into the contrastive learning process.
• Employs gradient-based saliency maps to interpret model attention and validate focus on facial skin regions.
• Trains end-to-end using contrastive learning with data augmentations, enabling self-supervision and label efficiency.

Figure 1: Illustration of rPPG spatial similarity. The rPPG signals from four facial areas (A, B, C, D) have similar waveforms and power spectrum densities (PSDs).

Experimental results

Research questions

• RQ1 Can rPPG be accurately estimated without any ground truth physiological signals using only video data and rPPG priors?
• RQ2 How does the model perform when trained with partially labeled or misaligned GT signals compared to fully supervised baselines?
• RQ3 To what extent does incorporating rPPG-specific priors improve robustness to noise and motion artifacts?
• RQ4 Can the model generalize well to unseen datasets and video conditions without fine-tuning?
• RQ5 How does the contrastive learning objective enhance signal quality and model interpretability?

Key findings

• Contrast-Phys+ achieves a mean absolute error (MAE) of 0.64 bpm on UBFC-rPPG with no injected noise, outperforming Gideon2021 (1.85 bpm) under the same conditions.
• With periodic noise injected into the video, Contrast-Phys+ maintains an MAE of 0.74 bpm, while Gideon2021 degrades to 22.47 bpm, demonstrating superior noise robustness.
• Even with 0% labeled data, Contrast-Phys+ achieves a Pearson correlation of 0.991 with GT signals, surpassing fully supervised methods in some settings.
• The model’s rPPG waveforms become increasingly aligned with ground truth as label ratio increases, with 100% labels yielding near-perfect peak detection.
• Saliency maps confirm that Contrast-Phys+ consistently focuses on facial skin regions, while baselines like Gideon2021 are distracted by noise or motion artifacts.
• The method generalizes well across five public datasets, including RGB and near-infrared videos, showing strong cross-dataset robustness.

Figure 2: Illustration of rPPG temporal similarity. The rPPG signals from two temporal windows (A, B) have similar PSDs.