[Paper Review] BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements

Girish Narayanswamy, Yujia Liu+5 more|arXiv (Cornell University)|Mar 21, 2023

Optical Imaging and Spectroscopy TechniquesMedicine3 citations

TL;DR

BigSmall proposes a unified, efficient multi-task deep learning architecture for joint remote measurement of facial actions, cardiac activity, and respiration from video. By combining a high-resolution 'Big' branch for spatial details and a low-resolution 'Small' branch with Wrapping Temporal Shift Modules (WTSM) for temporal dynamics, it achieves state-of-the-art accuracy with over 60% computational efficiency gains compared to task-specific models.

ABSTRACT

Understanding of human visual perception has historically inspired the design of computer vision architectures. As an example, perception occurs at different scales both spatially and temporally, suggesting that the extraction of salient visual information may be made more effective by paying attention to specific features at varying scales. Visual changes in the body due to physiological processes also occur at different scales and with modality-specific characteristic properties. Inspired by this, we present BigSmall, an efficient architecture for physiological and behavioral measurement. We present the first joint camera-based facial action, cardiac, and pulmonary measurement model. We propose a multi-branch network with wrapping temporal shift modules that yields both accuracy and efficiency gains. We observe that fusing low-level features leads to suboptimal performance, but that fusing high level features enables efficiency gains with negligible loss in accuracy. Experimental results demonstrate that BigSmall significantly reduces the computational costs. Furthermore, compared to existing task-specific models, BigSmall achieves comparable or better results on multiple physiological measurement tasks simultaneously with a unified model.

Motivation & Objective

• To develop a unified deep learning model that jointly predicts disparate physiological signals—facial actions, cardiac activity, and respiration—using only video input.
• To address the challenge of differing spatial and temporal scales across physiological signals, which hinders efficient multi-task learning.
• To reduce computational cost while maintaining or improving accuracy compared to task-optimized models.
• To explore feature fusion strategies that preserve performance while enhancing efficiency.
• To enable real-time, on-device deployment of multi-modal physiological sensing through efficient architecture design.

Proposed method

• The model uses a dual-branch architecture: a 'Big' branch with high-resolution input for capturing fine spatial features of facial actions, and a 'Small' branch with low-resolution input to compress spatial noise and emphasize temporal dynamics.
• The 'Small' branch employs Wrapping Temporal Shift Modules (WTSM), which shift temporal features across frames in a cyclic manner to preserve temporal context even with limited frames, avoiding the issue of zero-padded features.
• WTSM is specifically designed for low-frame-count scenarios, where standard zero-padded temporal shift modules degrade performance due to excessive zero values.
• Feature fusion is applied at high-level representations rather than low-level features, which empirically yields better accuracy with minimal computational overhead.
• The model is trained end-to-end on a unified loss combining task-specific objectives for facial action units (AUs), photoplethysmography (PPG), and respiration.
• Mixed spatial and temporal scales are used to optimize computational efficiency, reducing FLOPs by over 60% compared to baseline multi-task models.

Experimental results

Research questions

• RQ1 Can a single deep learning model effectively and efficiently predict multiple physiological signals with disparate spatial and temporal characteristics from video?
• RQ2 Does fusing features at high-level representations rather than low-level representations yield better accuracy and efficiency in multi-task physiological measurement?
• RQ3 Can a Wrapping Temporal Shift Module (WTSM) outperform standard temporal shift modules in low-frame-count scenarios typical of remote physiological sensing?
• RQ4 To what extent can a unified model generalize across facial actions, pulse, and respiration without fine-tuning, compared to task-optimized models?
• RQ5 How much computational efficiency can be gained by combining dual-branch design with mixed spatial-temporal scale processing in multi-task physiological video analysis?

Key findings

• BigSmall achieves state-of-the-art accuracy on all three tasks—facial action unit recognition, pulse rate estimation, and respiration rate estimation—while reducing computational cost by over 60% compared to baseline multi-task models.
• Fusing features at high-level representations leads to negligible accuracy loss compared to low-level fusion, but enables significant efficiency gains.
• The Wrapping Temporal Shift Module (WTSM) outperforms standard zero-padded temporal shift modules in low-frame scenarios, where the latter suffer from excessive zeroed features.
• Pre-training on one modality and fine-tuning on others yields suboptimal performance, highlighting the need for a unified multi-task architecture like BigSmall.
• The model demonstrates strong generalization across diverse physiological signals using only video input, with no need for additional sensors or modality-specific architectures.