[Paper Review] Efficient Mixture-of-Expert for Video-based Driver State and Physiological Multi-task Estimation in Conditional Autonomous Driving

Jiyao Wang, Xiao Yang+5 more|arXiv (Cornell University)|Oct 28, 2024

Sleep and Work-Related FatiguePsychology3 citations

TL;DR

This paper proposes VDMoE, a computationally efficient Mixture-of-Experts model that leverages RGB video and remote photoplethysmography (rPPG) to jointly estimate driver drowsiness, cognitive load, heart rate, and respiratory rate in SAE Level-2/3 autonomous driving. By using facial keypoint features and a prior-inclusive regularization loss, VDMoE achieves high accuracy while minimizing computational cost, validated on a new dataset (MCDD) and public benchmarks.

ABSTRACT

Road safety remains a critical challenge worldwide, with approximately 1.35 million fatalities annually attributed to traffic accidents, often due to human errors. As we advance towards higher levels of vehicle automation, challenges still exist, as driving with automation can cognitively over-demand drivers if they engage in non-driving-related tasks (NDRTs), or lead to drowsiness if driving was the sole task. This calls for the urgent need for an effective Driver Monitoring System (DMS) that can evaluate cognitive load and drowsiness in SAE Level-2/3 autonomous driving contexts. In this study, we propose a novel multi-task DMS, termed VDMoE, which leverages RGB video input to monitor driver states non-invasively. By utilizing key facial features to minimize computational load and integrating remote Photoplethysmography (rPPG) for physiological insights, our approach enhances detection accuracy while maintaining efficiency. Additionally, we optimize the Mixture-of-Experts (MoE) framework to accommodate multi-modal inputs and improve performance across different tasks. A novel prior-inclusive regularization method is introduced to align model outputs with statistical priors, thus accelerating convergence and mitigating overfitting risks. We validate our method with the creation of a new dataset (MCDD), which comprises RGB video and physiological indicators from 42 participants, and two public datasets. Our findings demonstrate the effectiveness of VDMoE in monitoring driver states, contributing to safer autonomous driving systems. The code and data will be released.

Motivation & Objective

• Address the critical need for real-time, non-invasive driver monitoring in SAE Level-2/3 autonomous vehicles to prevent accidents caused by drowsiness or cognitive overload.
• Overcome limitations of existing video-based DMS that rely on full-frame inputs (high cost) or single-task estimation (inadequate for real-world complexity).
• Develop a multi-task, efficient deep learning framework that jointly estimates drowsiness, cognitive load, heart rate, and respiratory rate from facial video and rPPG signals.
• Improve model robustness and convergence by integrating statistical priors into the loss function to align predictions with known physiological and behavioral patterns.
• Create and release a new large-scale, multi-modal dataset (MCDD) to support research in conditional autonomous driving scenarios with co-occurring cognitive and physiological states.

Proposed method

• Propose VDMoE, a multi-task Mixture-of-Experts (MoE) architecture using lightweight Multi-Layer Perceptrons (MLPs) as experts, enabling efficient computation and dynamic routing per input.
• Use facial keypoint features (e.g., eye and mouth regions) as input to reduce computational load while preserving temporal and spatial dynamics relevant to drowsiness and cognitive state.
• Incorporate remote photoplethysmography (rPPG) signals extracted from facial video to estimate physiological indicators like heart rate and respiratory rate.
• Introduce a novel prior-inclusive regularization loss, $\mathcal{L}_{\text{align}}$, which aligns model outputs with known statistical priors (e.g., expected ranges of HR/RR), improving convergence and reducing overfitting.
• Design the MoE gating mechanism to dynamically route inputs to task-specific experts, enabling specialized processing per output head (drowsiness, cognitive load, HR, RR).
• Train the model end-to-end using a combination of task-specific losses and the prior-aligned loss, ensuring both high accuracy and generalization across diverse driving conditions.

Experimental results

Research questions

• RQ1 Can a lightweight, multi-task MoE model effectively estimate multiple driver states (drowsiness, cognitive load) and physiological indicators (HR, RR) from RGB video and rPPG signals in a single inference pass?
• RQ2 How does incorporating statistical priors into the loss function improve model convergence and generalization in driver state estimation under limited data?
• RQ3 To what extent does using facial keypoint features instead of full-frame video reduce computational cost without sacrificing performance in multi-task driver monitoring?
• RQ4 How well does the proposed VDMoE model generalize across diverse driving conditions and individual differences, especially in the presence of environmental variability (lighting, motion, etc.)?
• RQ5 Can a newly collected, large-scale dataset (MCDD) that captures co-occurring cognitive load and drowsiness in simulated SAE Level-2/3 driving scenarios support more robust and realistic evaluation of multi-task DMS models?

Key findings

• VDMoE achieves state-of-the-art performance across all four tasks—drowsiness detection, cognitive load estimation, heart rate, and respiratory rate—on both the newly collected MCDD dataset and two public benchmarks.
• The model demonstrates significant computational efficiency by using facial keypoint features and MLP-based experts, reducing FLOPs compared to 3D-CNN or Transformer-based baselines while maintaining high accuracy.
• The prior-inclusive regularization loss $\mathcal{L}_{\text{align}}$ accelerates training convergence and reduces overfitting, especially in low-data regimes, by constraining predictions within biologically plausible ranges.
• The MCDD dataset, collected from 42 participants in a driving simulator with controlled task conditions, captures realistic co-occurrence of cognitive load and drowsiness, enabling more robust evaluation of multi-task DMS.
• Empirical results show that joint estimation of multiple driver states improves overall model performance compared to single-task baselines, indicating the benefit of multi-task inductive bias.
• The model generalizes well across diverse participants and driving scenarios, with consistent performance across different lighting and motion conditions in the simulator, suggesting practical deployability in real-world settings.