[Paper Review] Driver Action Prediction Using Deep (Bidirectional) Recurrent Neural Network

Oluwatobi Olabiyi, Eric Martinson+2 more|arXiv (Cornell University)|Jun 7, 2017

Human Pose and Action Recognition26 references71 citations

TL;DR

The paper formulates driver action prediction as a time-series anomaly detection problem and uses a deep bidirectional recurrent neural network to anticipate actions (acceleration, braking, lane change, turning) up to 5 seconds before execution by fusing camera/environment, driver, and vehicle dynamics data.

ABSTRACT

Advanced driver assistance systems (ADAS) can be significantly improved with effective driver action prediction (DAP). Predicting driver actions early and accurately can help mitigate the effects of potentially unsafe driving behaviors and avoid possible accidents. In this paper, we formulate driver action prediction as a timeseries anomaly prediction problem. While the anomaly (driver actions of interest) detection might be trivial in this context, finding patterns that consistently precede an anomaly requires searching for or extracting features across multi-modal sensory inputs. We present such a driver action prediction system, including a real-time data acquisition, processing and learning framework for predicting future or impending driver action. The proposed system incorporates camera-based knowledge of the driving environment and the driver themselves, in addition to traditional vehicle dynamics. It then uses a deep bidirectional recurrent neural network (DBRNN) to learn the correlation between sensory inputs and impending driver behavior achieving accurate and high horizon action prediction. The proposed system performs better than other existing systems on driver action prediction tasks and can accurately predict key driver actions including acceleration, braking, lane change and turning at durations of 5sec before the action is executed by the driver.

Motivation & Objective

• Improve ADAS through early and accurate driver action prediction.
• Leverage multi-modal sensory inputs (environment, driver, vehicle dynamics) for anticipation.
• Formulate driver action prediction as a time-series anomaly detection problem.
• Develop a real-time data acquisition, processing, and learning framework.

Proposed method

• Apply a deep bidirectional recurrent neural network (DBRNN) to learn correlations between sensory inputs and impending driver behavior.
• Integrate camera-based driving environment data with driver signals and traditional vehicle dynamics.
• Treat impending driver actions as anomalies to be predicted in a time-series framework.
• Implement a real-time data acquisition, processing, and learning pipeline for ongoing prediction.
• Evaluate the system on key actions—acceleration, braking, lane change, and turning—with a 5-second prediction horizon.

Experimental results

Research questions

• RQ1 Can a DBRNN effectively learn the correlation between multi-modal inputs and impending driver actions?
• RQ2 How early and accurately can driver actions be predicted using the proposed framework?
• RQ3 Does the DBRNN approach improve prediction for acceleration, braking, lane changes, and turning compared to existing systems?
• RQ4 What are the benefits and limitations of formulating driver action prediction as a time-series anomaly detection problem?

Key findings

• The DBRNN-based system learns correlations between sensory inputs and impending driver behavior.
• The framework operates in real time, integrating camera, driver, and vehicle dynamics data.
• The proposed approach outperforms existing systems on driver action prediction tasks.
• It can accurately predict key actions (acceleration, braking, lane change, turning) at a horizon of 5 seconds before execution.