[Paper Review] Neuron Structure Modeling for Generalizable Remote Physiological Measurement

Hao Lu, Zitong Yu+2 more|arXiv (Cornell University)|Mar 10, 2023

Non-Invasive Vital Sign Monitoring4 citations

TL;DR

This paper proposes NEST, a domain-label-free method for remote physiological measurement using rPPG that enhances generalization by modeling neural structure through coverage maximization, targeted alignment, and diversity maximization. It outperforms state-of-the-art methods on cross-dataset and intra-dataset benchmarks without relying on explicit domain labels, demonstrating robustness in real-world, agnostic domain settings.

ABSTRACT

Remote photoplethysmography (rPPG) technology has drawn increasing attention in recent years. It can extract Blood Volume Pulse (BVP) from facial videos, making many applications like health monitoring and emotional analysis more accessible. However, as the BVP signal is easily affected by environmental changes, existing methods struggle to generalize well for unseen domains. In this paper, we systematically address the domain shift problem in the rPPG measurement task. We show that most domain generalization methods do not work well in this problem, as domain labels are ambiguous in complicated environmental changes. In light of this, we propose a domain-label-free approach called NEuron STructure modeling (NEST). NEST improves the generalization capacity by maximizing the coverage of feature space during training, which reduces the chance for under-optimized feature activation during inference. Besides, NEST can also enrich and enhance domain invariant features across multi-domain. We create and benchmark a large-scale domain generalization protocol for the rPPG measurement task. Extensive experiments show that our approach outperforms the state-of-the-art methods on both cross-dataset and intra-dataset settings.

Motivation & Objective

• Address the challenge of domain shift in remote photoplethysmography (rPPG) due to environmental variations and lack of labeled domains.
• Overcome limitations of existing domain generalization methods that rely on clearly defined domain labels, which are ambiguous in complex rPPG scenarios.
• Develop a method that improves model generalization in unseen environments without requiring domain annotations.
• Create and benchmark a large-scale domain generalization protocol for rPPG to evaluate model robustness across diverse datasets.
• Enhance feature space coverage, suppress domain-specific features, and enrich discriminative features to improve out-of-distribution performance.

Proposed method

• Propose Neural Structure Coverage Maximization (NEST-CM) to regularize feature space by encouraging activation of under-optimized neural units during training.
• Introduce Neural Structure Targeted Alignment (NEST-TA) to align features of samples with similar physiological signals across domains, improving domain-invariant representation.
• Develop Neural Structure Diversity Maximization (NEST-DM) to push apart representations of different samples, enhancing discriminative capacity.
• Use spatial-temporal maps (STMaps) from facial videos as input to extract spatiotemporal features for rPPG signal estimation.
• Train the model using a combination of NEST-CM, NEST-TA, and NEST-DM losses to jointly optimize for robustness, invariance, and diversity.
• Apply the method in both cross-dataset (MSDG) and intra-dataset (5-fold CV on VIPL-HR) settings to validate generalization.

Experimental results

Research questions

• RQ1 Can a domain generalization method be effective in rPPG when domain labels are ambiguous or undefined due to environmental and device variations?
• RQ2 How can neural structure modeling improve feature space coverage and reduce abnormal activation in out-of-distribution samples?
• RQ3 To what extent can domain-invariant features be enhanced without relying on explicit domain annotations in rPPG?
• RQ4 Does the proposed method generalize better than existing SOTA methods on both cross-dataset and intra-dataset evaluation protocols?
• RQ5 How do hyperparameters in NEST-CM, NEST-TA, and NEST-DM affect model performance and robustness?

Key findings

• NEST achieves the best performance on the multi-domain generalization (MSDG) protocol, outperforming all SOTA methods including DeepPhys and TS-CAN.
• On the VIPL-HR dataset, NEST reduces MAE by 0.46 and RMSE by 0.56 compared to the baseline, achieving 4.76 MAE and 7.51 RMSE.
• NEST achieves an r-value of 0.84 on VIPL-HR, significantly outperforming the next-best method (Dual-GAN) with r=0.81.
• The method shows robustness in agnostic domain settings, where datasets cannot be clearly segmented into distinct domains, proving its label-free design is effective.
• Hyperparameter ablation shows optimal performance at ρ=0.1 for NEST-CM, σ=5 for NEST-TA, and τ with moderate values for NEST-DM, confirming the importance of balanced regularization.
• Pearson correlation analysis confirms that NEST-TA produces smoother representation transitions across continuous HR labels, reducing prediction errors for OOD samples.