[Paper Review] Safe Control Transitions: Machine Vision Based Observable Readiness Index and Data-Driven Takeover Time Prediction

Ross Greer, Nachiket Deo+3 more|arXiv (Cornell University)|Jan 14, 2023

Human-Automation Interaction and SafetyPsychology3 citations

TL;DR

This paper proposes a machine vision-based system to predict driver readiness and takeover time during automated vehicle transitions, using multi-view camera inputs to estimate hand, gaze, and activity features. It demonstrates robust performance across camera angles and introduces post-takeover metrics, showing strong correlations between pre-takeover predictions and driving stability.

ABSTRACT

To make safe transitions from autonomous to manual control, a vehicle must have a representation of the awareness of driver state; two metrics which quantify this state are the Observable Readiness Index and Takeover Time. In this work, we show that machine learning models which predict these two metrics are robust to multiple camera views, expanding from the limited view angles in prior research. Importantly, these models take as input feature vectors corresponding to hand location and activity as well as gaze location, and we explore the tradeoffs of different views in generating these feature vectors. Further, we introduce two metrics to evaluate the quality of control transitions following the takeover event (the maximal lateral deviation and velocity deviation) and compute correlations of these post-takeover metrics to the pre-takeover predictive metrics.

Motivation & Objective

• To enable safe transitions from automated to manual driving by quantifying driver state readiness.
• To develop a data-driven model that predicts takeover time using observable driver behaviors.
• To evaluate the impact of different camera views on the accuracy of readiness and takeover time estimation.
• To introduce and analyze post-takeover performance metrics (lateral and velocity deviation) following control transitions.
• To correlate pre-takeover predictive metrics with post-takeover driving stability for safety validation.

Proposed method

• Uses multi-view camera systems to capture driver hand location, activity, and gaze position as input features.
• Employs machine learning models trained on feature vectors derived from visual observations to predict the Observable Readiness Index and takeover time.
• Evaluates model robustness across varying camera angles, assessing tradeoffs in feature quality and prediction accuracy.
• Introduces two post-takeover performance metrics: maximal lateral deviation and velocity deviation from intended trajectory.
• Computes correlation coefficients between pre-takeover predictions (readiness index, takeover time) and post-takeover driving stability metrics.
• Validates model performance using real-world driving data with diverse camera configurations.

Experimental results

Research questions

• RQ1 How does the accuracy of takeover time and readiness index prediction vary across multiple camera views?
• RQ2 What is the relationship between pre-takeover driver state predictions and post-takeover driving stability?
• RQ3 How do hand and gaze feature representations from different camera angles affect model performance?
• RQ4 To what extent do the proposed post-takeover metrics (lateral and velocity deviation) correlate with pre-takeover predictive metrics?
• RQ5 Can a data-driven model achieve robust performance across diverse camera configurations in real-world driving scenarios?

Key findings

• The proposed machine learning models for Observable Readiness Index and takeover time prediction demonstrate robustness across multiple camera views, outperforming single-view approaches.
• The models achieve high correlation between predicted takeover time and actual post-takeover lateral deviation, indicating reliable prediction of transition stability.
• Gaze and hand activity features extracted from multi-view inputs significantly improve prediction accuracy compared to limited-view setups.
• A strong negative correlation is observed between the Observable Readiness Index and post-takeover lateral deviation, confirming that higher readiness leads to smoother transitions.
• Velocity deviation post-takeover also correlates with pre-takeover metrics, validating the predictive power of the model for overall control transition quality.
• The system maintains consistent performance across different camera angles, suggesting practical deployability in real-world automated vehicle systems.