[Paper Review] Vision Transformers and YoloV5 based Driver Drowsiness Detection Framework

Ghanta Sai Krishna, Kundrapu Supriya+2 more|arXiv (Cornell University)|Sep 3, 2022

Sleep and Work-Related Fatigue19 citations

TL;DR

This paper proposes a novel driver drowsiness detection framework combining YOLOv5 for face detection and Vision Transformers (ViT) for binary classification of drowsiness. Trained on the UTA-RLDD dataset, the ViT model achieved 97.4% validation accuracy, and on a custom dataset of 39 participants across varied lighting conditions, it reached 95.5% accuracy, demonstrating strong real-world applicability in smart transportation systems.

ABSTRACT

Human drivers have distinct driving techniques, knowledge, and sentiments due to unique driving traits. Driver drowsiness has been a serious issue endangering road safety; therefore, it is essential to design an effective drowsiness detection algorithm to bypass road accidents. Miscellaneous research efforts have been approached the problem of detecting anomalous human driver behaviour to examine the frontal face of the driver and automobile dynamics via computer vision techniques. Still, the conventional methods cannot capture complicated driver behaviour features. However, with the origin of deep learning architectures, a substantial amount of research has also been executed to analyze and recognize driver's drowsiness using neural network algorithms. This paper introduces a novel framework based on vision transformers and YoloV5 architectures for driver drowsiness recognition. A custom YoloV5 pre-trained architecture is proposed for face extraction with the aim of extracting Region of Interest (ROI). Owing to the limitations of previous architectures, this paper introduces vision transformers for binary image classification which is trained and validated on a public dataset UTA-RLDD. The model had achieved 96.2\% and 97.4\% as it's training and validation accuracies respectively. For the further evaluation, proposed framework is tested on a custom dataset of 39 participants in various light circumstances and achieved 95.5\% accuracy. The conducted experimentations revealed the significant potential of our framework for practical applications in smart transportation systems.

Motivation & Objective

• To address the rising incidence of drowsy driving, which causes thousands of annual fatalities and injuries.
• To overcome limitations of conventional methods that fail to capture complex driver behavior features.
• To develop a robust, real-time drowsiness detection system using deep learning architectures.
• To improve performance over existing CNN-based and traditional computer vision methods in face detection and classification.
• To validate the framework on a diverse, real-world custom dataset under varying lighting and occlusion conditions.

Proposed method

• Fine-tuned a custom YOLOv5 model for accurate face detection and Region of Interest (ROI) extraction from driver frontal views.
• Applied data augmentation techniques to enhance model generalization across diverse lighting and facial occlusion scenarios.
• Designed a Vision Transformer (ViT) architecture for binary image classification of drowsy vs. non-drowsy states.
• Trained and validated the ViT model on the public UTA-RLDD dataset, achieving 96.2% training and 97.4% validation accuracy.
• Evaluated the full pipeline on a custom dataset of 39 participants collected across day, evening, and night conditions with varying facial coverings.
• Used standard metrics including precision, sensitivity, and F1-score to assess model performance.

Experimental results

Research questions

• RQ1 Can a YOLOv5-based face detection system effectively extract relevant facial regions for drowsiness analysis in real-time?
• RQ2 How does a Vision Transformer-based classifier compare to conventional CNNs in classifying driver drowsiness from facial images?
• RQ3 What is the performance of the proposed framework on a real-world custom dataset with variable lighting and occlusion conditions?
• RQ4 Does the integration of YOLOv5 and ViT yield superior accuracy and robustness compared to existing drowsiness detection models?
• RQ5 How do lighting conditions (day, evening, night) affect the classification accuracy of the ViT model?

Key findings

• The Vision Transformer model achieved 97.4% validation accuracy on the UTA-RLDD public dataset, indicating strong generalization on standard benchmarks.
• On a custom dataset of 39 participants, the ViT model reached 95.5% test accuracy, demonstrating strong real-world performance.
• The model performed best in evening conditions (97.8% accuracy), attributed to optimal lighting, and worst at night (92.8% accuracy), due to low-light challenges.
• The model achieved high precision (0.97), sensitivity (0.98), and F1-score (0.97), indicating robust and balanced classification performance.
• The YOLOv5 face detection component achieved approximately 95% mAP, confirming effective ROI localization.
• The framework outperforms many existing methods based on CNNs, GANs, and traditional computer vision, particularly in complex real-world scenarios.