[Paper Review] End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies

Hesham M. Eraqi, Mohamed Moustafa+1 more|arXiv (Cornell University)|Oct 10, 2017

Autonomous Vehicle Technology and Safety20 references70 citations

TL;DR

The paper proposes a Convolutional Long Short-Term Memory (C-LSTM) model that learns visual and temporal driving cues end-to-end, framing steering as a sinusoidal classification task; it achieves 35% RMSE improvement and 87% stability on the comma.ai dataset.

ABSTRACT

Steering a car through traffic is a complex task that is difficult to cast into algorithms. Therefore, researchers turn to training artificial neural networks from front-facing camera data stream along with the associated steering angles. Nevertheless, most existing solutions consider only the visual camera frames as input, thus ignoring the temporal relationship between frames. In this work, we propose a Convolutional Long Short-Term Memory Recurrent Neural Network (C-LSTM), that is end-to-end trainable, to learn both visual and dynamic temporal dependencies of driving. Additionally, We introduce posing the steering angle regression problem as classification while imposing a spatial relationship between the output layer neurons. Such method is based on learning a sinusoidal function that encodes steering angles. To train and validate our proposed methods, we used the publicly available Comma.ai dataset. Our solution improved steering root mean square error by 35% over recent methods, and led to a more stable steering by 87%.

Motivation & Objective

• Motivate steering as an end-to-end learning task from front-facing camera data.
• Incorporate temporal dependencies across frames via a C-LSTM architecture.
• Pose steering regression as a sinusoidal classification problem to exploit output topology.
• Show that sinusoidal classification with LSTM improves accuracy and stability over direct regression and previous classifiers.

Proposed method

• Use a CNN (pre-trained on ImageNet) to extract frame-level features.
• Process sequential CNN features with a stack of LSTM layers to model temporal dynamics.
• Encode steering angles as a sine wave across N output neurons to impose spatial correlation.
• Train end-to-end with a sliding window to learn mappings from frames X_{t-w}...X_t to steering angle Y_t.
• Decode steering angle by fitting a sine wave to the network outputs via least-squares during deployment.

Experimental results

Research questions

• RQ1 Can integrating temporal dependencies via a C-LSTM improve steering predictions from monocular video?
• RQ2 Does encoding the steering angle as a sinusoidal classification with spatially correlated outputs outperform direct regression and standard classification approaches?
• RQ3 What is the impact of sliding window length and network depth on prediction accuracy and stability?

Key findings

• C-LSTM improves steering RMSE to 16.01° with whiteness 9.7°/time unit versus CNN-based regression at 17.77° RMSE and 39.1 whiteness.
• Sinusoidal classification with sine wave fitting yields RMSE 14.93° and whiteness 8.2°, outperforming both direct regression and traditional classification.
• Compared to state-of-the-art methods, the proposed approach achieves a 35% reduction in steering RMSE and an 87% improvement in stability on the comma.ai dataset.
• On the evaluated table, the sine-wave classification with C-LSTM outperforms a baseline regression and a standard classification approach.
• The approach demonstrates that encoding regression as a correlated output sinusoids can improve end-to-end driving performance.