[Paper Review] Driver Intention Anticipation Based on In-Cabin and Driving Scene Monitoring

Yao Rong, Zeynep Akata+1 more|arXiv (Cornell University)|Jun 20, 2020

Autonomous Vehicle Technology and Safety18 references4 citations

TL;DR

This paper proposes an end-to-end deep learning framework that jointly models in-cabin driver behavior and out-cabin traffic scene motion using a ConvLSTM-based auto-encoder for motion feature extraction and a novel joint classifier. The method achieves 83.98% accuracy and 84.30% F1-score on the Brain4cars dataset, demonstrating that in-cabin and out-cabin features are complementary and significantly improve maneuver intention anticipation over single-stream approaches.

ABSTRACT

Numerous car accidents are caused by improper driving maneuvers. Serious injuries are however avoidable if such driving maneuvers are detected beforehand and the driver is assisted accordingly. In fact, various recent research has focused on the automated prediction of driving maneuver based on hand-crafted features extracted mainly from in-cabin driver videos. Since the outside view from the traffic scene may also contain informative features for driving maneuver prediction, we present a framework for the detection of the drivers' intention based on both in-cabin and traffic scene videos. More specifically, we (1) propose a Convolutional-LSTM (ConvLSTM)-based auto-encoder to extract motion features from the out-cabin traffic, (2) train a classifier which considers motions from both in- and outside of the cabin jointly for maneuver intention anticipation, (3) experimentally prove that the in- and outside image features have complementary information. Our evaluation based on the publicly available dataset Brain4cars shows that our framework achieves a prediction with the accuracy of 83.98% and F1-score of 84.3%.

Motivation & Objective

• To improve driver maneuver intention prediction by integrating both in-cabin driver behavior and out-cabin traffic scene video data.
• To develop an end-to-end learning framework that avoids manual encoding of traffic scene features, which is impractical for real-world deployment.
• To extract meaningful motion features from out-cabin videos using a ConvLSTM-based auto-encoder, enabling effective use of external scene information.
• To design a lightweight, parameter-efficient model suitable for resource-constrained mobile or in-vehicle platforms.
• To validate that in-cabin and out-cabin video streams provide complementary and mutually reinforcing information for intention prediction.

Proposed method

• A ConvLSTM-based auto-encoder is used to extract motion features from optical flow representations of consecutive out-cabin video frames.
• 3D ResNet-50 is employed to extract spatiotemporal features from in-cabin driver videos, capturing head movements and gestures.
• A novel classifier architecture integrates a motion decoder for outside features and jointly processes in-cabin and out-cabin features in an end-to-end trainable manner.
• The framework is trained using a balanced loss function to handle class imbalance across five maneuver types (e.g., turn left, turn right, change lane, etc.).
• The model is evaluated using 5-fold cross-validation on the publicly available Brain4cars dataset, with ablation studies on single- and dual-stream inputs.
• No hand-crafted or manually encoded features are used; all features are learned end-to-end from raw video data.

Experimental results

Research questions

• RQ1 Can motion features extracted from out-cabin traffic videos improve driver maneuver intention prediction when combined with in-cabin features?
• RQ2 Does a joint modeling approach using both in-cabin and out-cabin video streams yield better performance than single-stream baselines?
• RQ3 Can a lightweight, end-to-end deep learning framework effectively learn from limited real-world driving data without manual feature engineering?
• RQ4 Is the information from out-cabin videos complementary to in-cabin features, or does it degrade performance when fused?
• RQ5 Can the proposed framework achieve high accuracy with fewer parameters than prior large-scale models?

Key findings

• The proposed dual-stream framework achieves 83.98% accuracy and 84.30% F1-score on the Brain4cars dataset, significantly outperforming single-stream baselines.
• Using only out-cabin videos, the model achieves 60.87% accuracy and 66.38% F1-score, demonstrating that traffic scene motion contains valuable predictive signals.
• The model surpasses a prior state-of-the-art method using 3D ResNet-101 by 8.48 percentage points in accuracy and 11.10 percentage points in F1-score, despite using fewer parameters.
• The framework does not require threshold-based prediction policies, as it maintains high confidence across all maneuver classes without over-reliance on thresholding.
• The results confirm that in-cabin and out-cabin features are complementary, with joint modeling yielding superior performance compared to isolated streams.
• The model is parameter-efficient, with only 57.92 million parameters when using both streams, making it suitable for deployment on mobile or embedded platforms.