[Paper Review] Real-Time Driver Monitoring Systems through Modality and View Analysis

Yiming Ma, Víctor Sánchez+4 more|arXiv (Cornell University)|Oct 17, 2022

Sleep and Work-Related Fatigue4 citations

TL;DR

This paper proposes real-time driver monitoring systems using single-image inputs from top-view infrared cameras, achieving state-of-the-art performance (97.5% AUC-PR) with significantly reduced computation by discarding temporal modeling. It demonstrates that top-view infrared modality alone outperforms multimodal video-based approaches in efficiency and accuracy, enabling deployable, low-latency inference on embedded devices.

ABSTRACT

Driver distractions are known to be the dominant cause of road accidents. While monitoring systems can detect non-driving-related activities and facilitate reducing the risks, they must be accurate and efficient to be applicable. Unfortunately, state-of-the-art methods prioritize accuracy while ignoring latency because they leverage cross-view and multimodal videos in which consecutive frames are highly similar. Thus, in this paper, we pursue time-effective detection models by neglecting the temporal relation between video frames and investigate the importance of each sensing modality in detecting drives' activities. Experiments demonstrate that 1) our proposed algorithms are real-time and can achieve similar performances (97.5\% AUC-PR) with significantly reduced computation compared with video-based models; 2) the top view with the infrared channel is more informative than any other single modality. Furthermore, we enhance the DAD dataset by manually annotating its test set to enable multiclassification. We also thoroughly analyze the influence of visual sensor types and their placements on the prediction of each class. The code and the new labels will be released.

Motivation & Objective

• To develop real-time, computationally efficient driver monitoring systems (DMS) that can be deployed on embedded automotive hardware.
• To investigate the relative importance of different visual modalities (RGB, depth, IR) and camera placements (top, front) for detecting non-driving-related activities (NDRAs).
• To enhance the DAD dataset by manually annotating its test set for multiclass classification of driver activities.
• To evaluate the trade-off between accuracy and inference efficiency in image-based vs. video-based DMS architectures.
• To identify the most informative single modality and camera configuration for real-time DMS deployment.

Proposed method

• The authors use single-frame images instead of video clips to eliminate temporal modeling, drastically reducing FLOPs and inference time.
• They evaluate multiple modality combinations (RGB, depth, IR) across different camera views (top, front) using feature-level and decision-level fusion strategies.
• A ResNet-18 and MobileNet-V2 backbone are used for feature extraction, with models trained on the DAD dataset for binary and multiclass classification.
• The study compares inference efficiency across platforms (ARM64 and AMD64) to validate real-time performance under embedded constraints.
• The DAD test set is manually re-labeled with specific activity classes to enable multiclass evaluation and class-wise analysis.
• Performance is evaluated using AUC-PR and AUC-ROC, with ablation studies on modality and view combinations.

Experimental results

Research questions

• RQ1 Can image-based models achieve comparable accuracy to video-based models while enabling real-time inference in embedded systems?
• RQ2 Which visual modality (RGB, depth, IR) and camera view (top, front) provide the most informative features for driver activity recognition?
• RQ3 How does the omission of temporal modeling affect detection performance in driver monitoring systems?
• RQ4 What is the impact of data imbalance and backbone capacity on multiclass classification performance?
• RQ5 Can a single, optimized modality and view combination outperform multimodal, video-based approaches in both accuracy and efficiency?

Key findings

• Image-based models achieve 97.5% AUC-PR and 95.6% AUC-ROC, matching state-of-the-art video-based models while reducing FLOPs by up to 10x.
• The top-view infrared (IR) modality alone achieves the highest performance (97.1% AUC-PR) among all tested single-modal configurations.
• Inference times for image-based models are under 0.03 seconds on ARM64 (M1 Pro), meeting real-time requirements (≤0.3556s per frame) for 45 Hz video streams.
• Feature-level fusion outperforms decision-level fusion when combining two views, with (Top+Front) IR achieving 96.7% AUC-PR.
• The top view with IR modality significantly reduces confusion between phone-related distractions and normal driving, indicating high discriminative power.
• Despite strong binary performance, multiclass classification remains challenging due to data imbalance and limited representation capacity of ResNet-18.