[Paper Review] Two-stream convolutional networks for end-to-end learning of self-driving cars

Nelson Fernández|arXiv (Cornell University)|Nov 13, 2018

Autonomous Vehicle Technology and Safety13 references10 citations

TL;DR

This paper proposes a two-stream 2D-CNN architecture that jointly learns spatial features from raw images and temporal features from optical flow to improve end-to-end self-driving car steering prediction. By using multitask learning with a previous steering angle as an auxiliary task, the model achieves a 30% improvement in accuracy and stability over baseline regression methods on the Comma.ai dataset, with whiteness reduced to 4.97 degrees/time—closer to human driving behavior.

ABSTRACT

We propose a methodology to extend the concept of Two-Stream Convolutional Networks to perform end-to-end learning for self-driving cars with temporal cues. The system has the ability to learn spatiotemporal features by simultaneously mapping raw images and pre-calculated optical flows directly to steering commands. Although optical flows encode temporal-rich information, we found that 2D-CNNs are prone to capturing features only as spatial representations. We show how the use of Multitask Learning favors the learning of temporal features via inductive transfer from a shared spatiotemporal representation. Preliminary results demonstrate a competitive improvement of 30% in prediction accuracy and stability compared to widely used regression methods trained on the Comma.ai dataset.

Motivation & Objective

• To address the limitation of single-frame regression in autonomous driving by incorporating temporal dynamics.
• To investigate whether 2D-CNNs can effectively learn temporal features from optical flow for end-to-end steering prediction.
• To evaluate the effectiveness of multitask learning in improving temporal generalization through inductive transfer.
• To benchmark the proposed two-stream architecture against state-of-the-art 2D-CNN regression models on the Comma.ai dataset.
• To demonstrate that shared spatiotemporal representations via element-wise multiplication enhance feature fusion and prediction stability.

Proposed method

• The model uses two identical 2D-CNN branches: one processing raw RGB images for spatial features, and another processing precomputed optical flow for temporal motion cues.
• Global average pooling is applied to each stream’s final convolutional layer to reduce parameters and prevent overfitting.
• The two stream embeddings are fused via element-wise multiplication to create a shared spatiotemporal representation.
• A multilayer perceptron (MLP) with linear output performs regression to predict the current steering angle.
• Multitask learning is employed by training the network to predict both the current and previous steering angles, with the latter used only during training.
• A two-stage training strategy is used: first, each stream is pre-trained independently; second, the full network is fine-tuned with a focus on the final regression head.

Experimental results

Research questions

• RQ1 Can 2D-CNNs effectively learn temporal features from optical flow representations in end-to-end self-driving systems?
• RQ2 Does the fusion of spatial and temporal streams via a shared spatiotemporal representation improve regression accuracy and stability?
• RQ3 To what extent does multitask learning with a previous steering angle as an auxiliary task enhance temporal generalization?
• RQ4 How does the proposed two-stream architecture compare in performance to single-stream 2D-CNNs trained on raw images or optical flows alone?
• RQ5 Can the use of optical flow and multitask learning reduce prediction bias and improve whiteness (temporal smoothness) in steering commands?

Key findings

• The Two-Stream Convolutional Network achieved an RMSE of 12.52 degrees on the Comma.ai test set, representing a 30% improvement in accuracy over baseline regression methods.
• The model’s whiteness was reduced to 4.97 degrees/time, significantly closer to the human driver’s whiteness of 4.36 degrees/time, indicating smoother steering predictions.
• Training on optical flows alone yielded a 30% accuracy gain over raw images (RMSE 14.21 vs. 20.55 for Simple-CNN), but did not improve whiteness, indicating spatial-only learning.
• The proposed architecture outperformed state-of-the-art models including Comma.ai’s 2D-CNN (RMSE 23.99) and Nvidia PilotNet (RMSE 20.55).
• The use of multitask learning with a previous steering angle as an auxiliary task enabled inductive transfer that corrected temporal bias and improved stability.
• The scatter plot analysis confirmed that the two-stream model reduced bias in both spatial and temporal dimensions compared to single-stream models.