[Paper Review] Deep-HR: Fast Heart Rate Estimation from Face Video Under Realistic Conditions

Mohammad Sabokrou, Masoud Pourreza+3 more|arXiv (Cornell University)|Feb 12, 2020

Non-Invasive Vital Sign Monitoring24 references4 citations

TL;DR

This paper proposes Deep-HR, a deep learning-based method for fast and accurate remote heart rate estimation from face videos under realistic conditions. By decoupling feature extraction (Front-End) from regression (Back-End) and using adversarially trained autoencoders to refine low-quality video frames, it achieves state-of-the-art performance with 3.41 RMSE on MAHNOB and 6.58 RMSE on the new HR-D dataset, while running at 100 FPS on a GPU.

ABSTRACT

This paper presents a novel method for remote heart rate (HR) estimation. Recent studies have proved that blood pumping by the heart is highly correlated to the intense color of face pixels, and surprisingly can be utilized for remote HR estimation. Researchers successfully proposed several methods for this task, but making it work in realistic situations is still a challenging problem in computer vision community. Furthermore, learning to solve such a complex task on a dataset with very limited annotated samples is not reasonable. Consequently, researchers do not prefer to use the deep learning approaches for this problem. In this paper, we propose a simple yet efficient approach to benefit the advantages of the Deep Neural Network (DNN) by simplifying HR estimation from a complex task to learning from very correlated representation to HR. Inspired by previous work, we learn a component called Front-End (FE) to provide a discriminative representation of face videos, afterward a light deep regression auto-encoder as Back-End (BE) is learned to map the FE representation to HR. Regression task on the informative representation is simple and could be learned efficiently on limited training samples. Beside of this, to be more accurate and work well on low-quality videos, two deep encoder-decoder networks are trained to refine the output of FE. We also introduce a challenging dataset (HR-D) to show that our method can efficiently work in realistic conditions. Experimental results on HR-D and MAHNOB datasets confirm that our method could run as a real-time method and estimate the average HR better than state-of-the-art ones.

Motivation & Objective

• To address the challenge of remote heart rate estimation in realistic, uncontrolled environments where traditional methods fail due to noise, motion, and illumination variations.
• To enable effective deep learning for HR estimation despite limited annotated training data, which hinders model training.
• To develop a real-time, computationally efficient method suitable for practical applications such as healthcare and fitness monitoring.
• To improve robustness against low-quality video inputs and facial obstructions (e.g., beards) through end-to-end refinement of intermediate representations.

Proposed method

• The method uses a two-stage deep learning framework: a Front-End (FE) that extracts discriminative, HR-correlated representations from face video frames without requiring HR-labeled data.
• A Back-End (BE) regression network maps the FE output to heart rate, trained only on limited labeled HR data, simplifying the learning task.
• Two adversarially trained encoder-decoder networks ($\mathcal{G}_1$ and $\mathcal{G}_2$) refine the FE output and the extracted color signals, improving robustness to noise and degradation.
• The FE is trained independently using self-supervised or unsupervised representation learning, focusing on capturing subtle skin color changes linked to blood volume changes.
• The BE is a lightweight fully connected network trained to regress HR from the FE features, enabling fast inference.
• The entire system is trained end-to-end with joint optimization of BE and refiners, leveraging generative adversarial learning to enhance signal quality.

Experimental results

Research questions

• RQ1 Can a deep learning-based approach achieve accurate heart rate estimation from face videos when only limited labeled training data is available?
• RQ2 How can deep neural networks be effectively trained for HR estimation despite the scarcity of annotated video data?
• RQ3 Can adversarial refinement networks improve the robustness of HR estimation under low-quality video conditions, such as noise, motion, or facial hair?
• RQ4 Is it possible to design a real-time, end-to-end deep learning system for remote HR estimation that outperforms traditional signal-processing pipelines?

Key findings

• Deep-HR achieves a test RMSE of 3.41 on the MAHNOB dataset, outperforming all prior state-of-the-art methods, which reported RMSEs ranging from 4.07 to 25.9.
• On the newly introduced HR-D dataset, the method achieves an RMSE of 6.58, significantly outperforming previous methods that reported RMSEs up to 24.71.
• The method runs at 100 frames per second on a GeForce GTX 1080 GPU, confirming its real-time capability for practical deployment.
• The adversarial refinement networks ($\mathcal{G}_1$ and $\mathcal{G}_2$) effectively reduce noise and restore degraded facial regions, improving signal quality and estimation accuracy.
• The model demonstrates robustness to facial obstructions such as beards and mustaches, which degrade performance in prior methods due to their sensitivity to pixel color changes in ROI.
• The method achieves a correlation coefficient ($r$) of 0.92 on MAHNOB, indicating strong alignment between predicted and ground-truth HR values.