[Paper Review] DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks

Weixuan Chen, Daniel McDuff|arXiv (Cornell University)|May 21, 2018

Non-Invasive Vital Sign Monitoring37 references44 citations

TL;DR

DeepPhys introduces an end-to-end convolutional attention network for non-contact heart rate and breathing rate estimation from RGB and infrared videos, using a skin-reflection-based motion representation and appearance-guided attention to robustly handle head motion and lighting variations.

ABSTRACT

Non-contact video-based physiological measurement has many applications in health care and human-computer interaction. Practical applications require measurements to be accurate even in the presence of large head rotations. We propose the first end-to-end system for video-based measurement of heart and breathing rate using a deep convolutional network. The system features a new motion representation based on a skin reflection model and a new attention mechanism using appearance information to guide motion estimation, both of which enable robust measurement under heterogeneous lighting and major motions. Our approach significantly outperforms all current state-of-the-art methods on both RGB and infrared video datasets. Furthermore, it allows spatial-temporal distributions of physiological signals to be visualized via the attention mechanism.

Motivation & Objective

• Develop an end-to-end deep learning system to recover heart rate (HR) and breathing rate (BR) from videos.
• Propose a new motion representation based on a skin reflection model suitable for heterogeneous illumination.
• Introduce an appearance-guided attention mechanism to robustly estimate physiological signals amid motion and lighting changes.
• Enable visualization of spatial-temporal distributions of physiological signals via the attention maps.

Proposed method

• Introduce a normalized frame-difference motion representation derived from a skin reflection model to capture physiological motions.
• Downsample frames and apply time-derivative normalization to remove stationary components and camera/luminance variations.
• Train a VGG-style CNN to map the motion representation to the derivative of the physiological signal p'(t) with MSE loss, followed by band-pass filtering and dominant frequency extraction for HR/BR.
• Incorporate an appearance model to learn soft-attention masks that guide the motion estimator by focusing on skin regions with stronger signals, using a 1x1 conv kernel and sigmoid-based normalization (can be applied to multiple layers).
• Use ensemble selection over training checkpoints to choose the model with the smallest frequency error for HR/BR estimation.
• Evaluate against several state-of-the-art methods on RGB and infrared datasets, under varying head motion and lighting conditions.

Experimental results

Research questions

• RQ1 Can an end-to-end CNN learn robust HR and BR signals directly from motion representations derived from skin reflection under diverse lighting?
• RQ2 Does appearance-guided attention improve motion estimation and physiological signal recovery over motion-only models?
• RQ3 How well does the proposed method generalize across datasets and scenarios with different devices, lighting, and head motion?
• RQ4 What is the impact of head motion on HR/BR estimation, and can the model maintain performance with large motions?

Key findings

• The CAN (convolutional attention network) outperforms state-of-the-art methods on multiple RGB and infrared datasets for both HR and BR estimation, especially under large head motions.
• The proposed normalized frame-difference motion representation combined with skin-reflection modeling enables robust physiological signal recovery under heterogeneous illumination.
• The appearance-guided attention mechanism improves ROI localization and boosts measurement accuracy compared to motion-only models.
• Attention visualizations reveal spatial-temporal distributions of physiological signals (e.g., HR-related regions) in video frames.
• Participant-dependent results show strong performance, with CAN achieving improved MAE and SNR over prior methods, particularly for high-motion tasks; participant-independent results also showing CAN advantages, though generally with lower absolute performance than the fully participant-dependent setting.
• Ensemble selection across training checkpoints further reduces frequency-domain errors for HR/BR estimation.