[Paper Review] FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensing

J.S. Joshi, Sos С. Agaian+1 more|arXiv (Cornell University)|Nov 3, 2024

Non-Invasive Vital Sign Monitoring4 citations

TL;DR

This paper proposes FactorizePhys, an end-to-end 3D-CNN architecture that uses a Factorized Self-Attention Module (FSAM) based on nonnegative matrix factorization (NMF) to jointly compute spatial, temporal, and channel attention in remote photoplethysmography (rPPG). The method achieves state-of-the-art performance on four public datasets with significantly fewer parameters and comparable latency, demonstrating superior cross-dataset generalization and robustness to occlusions.

ABSTRACT

Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.

Motivation & Objective

• To address the limitation of existing rPPG methods that compute attention independently across spatial, temporal, and channel dimensions.
• To develop a computationally efficient, joint multidimensional attention mechanism suitable for 3D video data in remote physiological sensing.
• To improve cross-dataset generalization and robustness to motion and occlusion in rPPG signal estimation.
• To demonstrate the versatility of the proposed attention module by integrating it into both 3D-CNN and 2D-CNN architectures.
• To provide a lightweight yet effective alternative to multi-head attention mechanisms in rPPG modeling.

Proposed method

• The Factorized Self-Attention Module (FSAM) applies nonnegative matrix factorization (NMF) to voxel embeddings to jointly model spatial, temporal, and channel attention.
• FSAM factorizes 3D feature tensors into low-rank components using NMF, enabling efficient and joint attention computation across all dimensions.
• The method is integrated into a 3D-CNN backbone (FactorizePhys) for end-to-end rPPG signal estimation from raw video frames.
• The same FSAM module is also deployed within a 2D-CNN-based rPPG architecture to validate its architectural versatility.
• The NMF-based attention mechanism is trained end-to-end with backpropagation, using a reconstruction loss to preserve relevant physiological signal patterns.
• Model efficiency is enhanced by reducing input spatial resolution, which lowers FLOPS while maintaining performance.

Experimental results

Research questions

• RQ1 Can nonnegative matrix factorization be effectively used to compute joint multidimensional attention in rPPG modeling?
• RQ2 Does the proposed FSAM module improve rPPG signal estimation performance compared to existing attention mechanisms?
• RQ3 How does the FactorizePhys architecture generalize across diverse datasets with varying conditions?
• RQ4 What is the impact of architectural choices and hyperparameters in the FSAM module on model performance?
• RQ5 Can the FSAM module be effectively transferred and applied to different CNN architectures, including 2D-CNNs?

Key findings

• FactorizePhys with FSAM outperforms all state-of-the-art rPPG methods across four public datasets in terms of correlation, RMSE, and MAE, with a significant margin.
• The proposed method achieves a model size of only 0.22 MB and 52,168 parameters, representing an order of magnitude reduction compared to existing SOTA models.
• Despite fewer parameters, the method maintains comparable inference latency, with 95.75ms on CPU and 8.97ms on GPU, outperforming models like PhysFormer and PhysNet.
• Ablation studies confirm that joint multidimensional attention via NMF is more effective than separate attention computation across dimensions.
• Visualization results show that FSAM learns more relevant spatial-temporal features, especially under occlusion conditions such as glasses, beards, and hair.
• The method demonstrates strong cross-dataset generalization, with consistent performance gains when trained on one dataset and tested on others, including iBVP, PURE, and UBFC-rPPG.