[Paper Review] Using image-extracted features to determine heart rate and blink duration for driver sleepiness detection

Erfan Darzi, Mohammadie-Zand, Armin+1 more|arXiv (Cornell University)|Nov 4, 2019

Non-Invasive Vital Sign Monitoring16 references4 citations

TL;DR

This paper proposes a non-invasive, remote driver sleepiness detection system using image-extracted features to estimate heart rate and blink duration. It employs facial video analysis with artificial neural networks (ANN) and linear discriminant analysis (LDA) for eye state classification, and independent component analysis (ICA) and chrominance-based methods for heart rate estimation, achieving 92% accuracy in blink detection and 13 BPM mean error in heart rate estimation.

ABSTRACT

Heart rate and blink duration are two vital physiological signals which give information about cardiac activity and consciousness. Monitoring these two signals is crucial for various applications such as driver drowsiness detection. As there are several problems posed by the conventional systems to be used for continuous, long-term monitoring, a remote blink and ECG monitoring system can be used as an alternative. For estimating the blink duration, two strategies are used. In the first approach, pictures of open and closed eyes are fed into an Artificial Neural Network (ANN) to decide whether the eyes are open or close. In the second approach, they are classified and labeled using Linear Discriminant Analysis (LDA). The labeled images are then be used to determine the blink duration. For heart rate variability, two strategies are used to evaluate the passing blood volume: Independent Component Analysis (ICA); and a chrominance based method. Eye recognition yielded 78-92% accuracy in classifying open/closed eyes with ANN and 71-91% accuracy with LDA. Heart rate evaluations had a mean loss of around 16 Beats Per Minute (BPM) for the ICA strategy and 13 BPM for the chrominance based technique.

Motivation & Objective

• To develop a remote, non-invasive system for detecting driver sleepiness using image-based physiological signals.
• To overcome limitations of conventional wearable sensors by using video-based monitoring for continuous, long-term assessment.
• To improve accuracy and reliability of sleepiness detection through fusion of blink duration and heart rate variability from facial images.
• To evaluate and compare multiple machine learning and signal processing techniques for eye state classification and heart rate estimation.
• To provide a practical, low-cost alternative to wearable devices for real-time driver monitoring systems.

Proposed method

• Captured facial video sequences to extract features related to eye state (open/closed) and blood volume changes.
• Used an Artificial Neural Network (ANN) to classify eye states from images, achieving 78–92% accuracy.
• Applied Linear Discriminant Analysis (LDA) as an alternative method for eye state classification, achieving 71–91% accuracy.
• Employed Independent Component Analysis (ICA) to estimate heart rate from facial video signals, with a mean error of ~16 BPM.
• Implemented a chrominance-based method to estimate heart rate variability, achieving a lower mean error of ~13 BPM.
• Combined blink duration estimation and heart rate variability to assess driver sleepiness in real time.

Experimental results

Research questions

• RQ1 Can image-extracted features from facial video accurately estimate blink duration for driver sleepiness detection?
• RQ2 How effective are ANN and LDA in classifying open and closed eye states from still images?
• RQ3 Which signal processing method—ICA or chrominance-based analysis—yields more accurate heart rate estimation from facial video?
• RQ4 What is the combined impact of blink duration and heart rate variability on sleepiness detection performance?
• RQ5 Can a non-contact, video-based system outperform conventional wearable sensors in long-term, continuous monitoring?

Key findings

• The ANN-based method achieved 78–92% accuracy in classifying open and closed eye states from facial images.
• The LDA-based method achieved 71–91% accuracy in eye state classification, demonstrating robustness across subjects.
• The ICA-based heart rate estimation method had a mean error of approximately 16 beats per minute (BPM).
• The chrominance-based heart rate estimation method achieved a lower mean error of around 13 BPM, outperforming ICA.
• The fusion of blink duration and heart rate variability from image-extracted features enables reliable, non-invasive sleepiness detection.
• The system demonstrates feasibility for real-time, remote monitoring of driver fatigue without wearable sensors.