[Paper Review] Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement

Xin Liu, Josh Fromm+2 more|arXiv (Cornell University)|Jun 6, 2020

Non-Invasive Vital Sign Monitoring38 references150 citations

TL;DR

The paper introduces MTTS-CAN, an on-device video-based method using temporal shift attention to jointly estimate heart rate and respiration in real time, achieving state-of-the-art accuracy with high FPS on ARM CPUs.

ABSTRACT

Telehealth and remote health monitoring have become increasingly important during the SARS-CoV-2 pandemic and it is widely expected that this will have a lasting impact on healthcare practices. These tools can help reduce the risk of exposing patients and medical staff to infection, make healthcare services more accessible, and allow providers to see more patients. However, objective measurement of vital signs is challenging without direct contact with a patient. We present a video-based and on-device optical cardiopulmonary vital sign measurement approach. It leverages a novel multi-task temporal shift convolutional attention network (MTTS-CAN) and enables real-time cardiovascular and respiratory measurements on mobile platforms. We evaluate our system on an Advanced RISC Machine (ARM) CPU and achieve state-of-the-art accuracy while running at over 150 frames per second which enables real-time applications. Systematic experimentation on large benchmark datasets reveals that our approach leads to substantial (20%-50%) reductions in error and generalizes well across datasets.

Motivation & Objective

• Motivate non-contact vital sign measurement for telehealth and remote monitoring, reducing infection risk and enabling accessible care.
• Develop an on-device, real-time model that jointly estimates cardiovascular and respiratory signals.
• Address privacy, portability, and precision constraints in camera-based vitals sensing.
• Demonstrate state-of-the-art accuracy and high inference speed on public datasets and ARM hardware.

Proposed method

• Propose MTTS-CAN, a multi-task temporal shift convolutional attention network for on-device vitals measurement.
• Incorporate temporal shift modules to model temporal dynamics without 3D convolutions and introduce an attention mechanism to focus on signal-bearing regions.
• Use a multi-task loss to jointly estimate blood volume pulse and respiration signals.
• Evaluate architectures against 2D-CAN, 3D-CAN, Hybrid-CAN baselines and on-device latency using TVM on an ARM-based Firefly-RK3399 platform.
• Ground optical modeling on Shafer’s Dichromatic Reflection Model to relate RGB video to BVP and respiration signals.
• Pre-process inputs with appearance and motion branches, including frame averaging for appearance and frame-difference based motion features.

Experimental results

Research questions

• RQ1 Can a temporal-shift based, on-device network accurately estimate pulse and respiration signals from facial videos?
• RQ2 Does integrating attention into a temporal-shift framework improve signal extraction under noise (motion, illumination) conditions?
• RQ3 How does multi-task sharing of intermediate representations between pulse and respiration affect accuracy and efficiency?
• RQ4 What are the on-device latency and memory requirements for MTTS-CAN on ARM hardware across different architectural variants?
• RQ5 How well do the proposed models generalize across datasets with different resolutions, backgrounds, and lighting?

Key findings

• MTTS-CAN delivers state-of-the-art accuracy for pulse measurement and respiration estimation on AFRL and MMSE-HR datasets.
• TS-CAN variants achieve faster inference (as low as 6-12 ms per frame) compared to baselines, enabling real-time performance.
• Multi-task TS-CAN (MTTS-CAN) reduces computation and memory by about 50% while maintaining accuracy relative to single-task models.
• Temporal models (MTTS-CAN, TS-CAN, Hybrid-CAN, 3D-CAN) outperform 2D-CAN and other baselines, particularly under higher head motion.
• Cross-dataset evaluation shows 25-50% error reduction over 2D-CAN, with MTTS-CAN and MT-Hybrid-CAN demonstrating strong generalization.