[论文解读] Facial Video-based Remote Physiological Measurement via Self-supervised Learning

Zijie Yue, Miaojing Shi+1 more|arXiv (Cornell University)|Oct 27, 2022

Non-Invasive Vital Sign Monitoring被引用 6

一句话总结

本文提出了一种自监督框架，用于从面部视频中进行远程生理测量，而无需依赖真实PPG信号。通过利用可学习的频率增强、局部rPPG专家聚合以及频率启发式损失，该方法在五个基准数据集上实现了心率、HRV和呼吸频率估计的最先进性能。

ABSTRACT

Facial video-based remote physiological measurement aims to estimate remote photoplethysmography (rPPG) signals from human face videos and then measure multiple vital signs (e.g. heart rate, respiration frequency) from rPPG signals. Recent approaches achieve it by training deep neural networks, which normally require abundant facial videos and synchronously recorded photoplethysmography (PPG) signals for supervision. However, the collection of these annotated corpora is not easy in practice. In this paper, we introduce a novel frequency-inspired self-supervised framework that learns to estimate rPPG signals from facial videos without the need of ground truth PPG signals. Given a video sample, we first augment it into multiple positive/negative samples which contain similar/dissimilar signal frequencies to the original one. Specifically, positive samples are generated using spatial augmentation. Negative samples are generated via a learnable frequency augmentation module, which performs non-linear signal frequency transformation on the input without excessively changing its visual appearance. Next, we introduce a local rPPG expert aggregation module to estimate rPPG signals from augmented samples. It encodes complementary pulsation information from different face regions and aggregate them into one rPPG prediction. Finally, we propose a series of frequency-inspired losses, i.e. frequency contrastive loss, frequency ratio consistency loss, and cross-video frequency agreement loss, for the optimization of estimated rPPG signals from multiple augmented video samples and across temporally neighboring video samples. We conduct rPPG-based heart rate, heart rate variability and respiration frequency estimation on four standard benchmarks. The experimental results demonstrate that our method improves the state of the art by a large margin.

研究动机与目标

• 为解决用于训练rPPG模型的同步面部视频与PPG信号数据集有限的问题。
• 实现在未配对的面部视频上进行自监督rPPG估计，而无需真实PPG信号。
• 提升模型在多样化受试者中的泛化能力和鲁棒性，尤其是在不同种族和光照条件下的表现。
• 设计一种频率感知的数据增强与损失函数，以增强从时间域和频域变化中学习rPPG信号的能力。

提出的方法

• 引入可学习频率增强（LFA）模块，非线性地转换面部视频中的信号频率，同时保持视觉外观不变。
• 应用空间增强（如旋转、翻转）以生成具有相同rPPG频率的正样本。
• 采用局部rPPG专家聚合（REA）模块，提取区域特定的搏动信号，并通过区域注意力和时空门控进行融合。
• 设计三种频率启发式损失：频率对比损失、频率比一致性损失和跨视频频率一致性损失，以优化rPPG估计。
• 仅使用面部视频片段，无需任何配对的PPG标注，以端到端方式自监督训练模型。
• 利用相邻视频片段作为额外正样本，以增强频率一致性与模型鲁棒性。

实验结果

研究问题

• RQ1 自监督学习能否在无需同步PPG真实信号的情况下有效训练rPPG模型？
• RQ2 与对比学习中固定比例重采样相比，可学习频率增强在提升模型泛化能力方面有何优势？
• RQ3 局部rPPG专家聚合模块在从异质面部区域提取信号方面有多大提升作用？
• RQ4 频率启发式损失在提升rPPG估计精度与一致性方面，对不同受试者和视频片段的改善程度如何？
• RQ5 在低信噪比场景（如深色皮肤）下，模型性能在不同种族群体中的表现如何变化，特别是对非洲裔群体？

主要发现

• 所提方法在五个标准rPPG基准数据集中均达到最先进性能，优于先前的监督与自监督方法。
• 在BP4D+数据集上，该方法在亚洲受试者中的心率估计平均绝对误差（MAE）为2.54，较Gideon等人[20]提升0.68 MAE。
• 对于非洲裔受试者，该方法的MAE为4.69，较Gideon等人[20]提升0.46，尽管由于深色皮肤在像素层面饱和，性能仍较低。
• 在高加索裔受试者中，该方法的MAE为2.89，显著优于先前最先进方法。
• 定性分析表明，信号周期性得到改善，且相邻R波间期变异减少，尤其在亚洲和高加索裔受试者中表现明显。
• 在平衡种族子集上的实验表明，非洲裔受试者性能下降主要源于相机对深色皮肤色调信号捕获能力的限制。