[Paper Review] Self-Driving Car Steering Angle Prediction: Let Transformer Be a Car Again

Chingis Oinar, Eunmin Kim|arXiv (Cornell University)|Apr 27, 2022

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes a novel Transformer-based architecture for self-driving car steering angle prediction, enhanced with optical flow features and multi-task learning. It achieves a private RMSE of 0.0577 and public RMSE of 0.0588 on the Udacity Self-Driving Car Challenge 2 dataset, outperforming baseline models and ranking 3rd–4th in the competition.

ABSTRACT

Self-driving vehicles are expected to be a massive economic influence over the coming decades. Udacity https://www.udacity.com/ has been working on a completely open-source self driving car. Thus, it regularly organizes various competitions, one of which was dedicated to steering angle prediction task. In this work, we perform an extensive study on this particular task by exploring the Udacity Self-driving Car Challenge 2. We provide insights on the previous teams' solutions. Moreover, we propose our new architecture that is inspired by some of the teams. We report our performance and compare it with multiple baseline architectures as well as other teams' solutions. We make our work available on GitHub and hope it is useful for the Udacity community and brings insights for future works https://github.com/chingisooinar/AI_self-driving-car

Motivation & Objective

• Improve upon existing deep learning models for steering angle prediction in autonomous driving.
• Address the challenge of predicting extreme steering angles, which are difficult for standard models.
• Explore the effectiveness of attention mechanisms and optical flow features in sequence modeling for driving tasks.
• Develop a model that generalizes well across diverse driving conditions, including varying weather, lighting, and road types.
• Outperform existing baselines and prior competition solutions on the Udacity Self-Driving Car Challenge 2 dataset.

Proposed method

• Adopt a Vision Transformer (ViT) backbone with a ResNet-18 feature extractor for RGB images.
• Introduce a separate branch to process optical flow images derived from consecutive RGB frames.
• Implement multi-task learning by jointly predicting steering angle and vehicle speed.
• Apply label smoothing with a factor of 0.35 to improve generalization and reduce overfitting.
• Train the model using mean squared error (MSE) loss on both steering angle and speed prediction heads.
• Use data augmentation and sequence modeling to capture temporal dependencies across frames.

Experimental results

Research questions

• RQ1 Can a Transformer-based architecture outperform CNN-LSTM and ResNet-based models in steering angle prediction?
• RQ2 How does incorporating optical flow features improve performance on extreme steering angle predictions?
• RQ3 To what extent does multi-task learning with speed prediction enhance the main steering angle regression task?
• RQ4 Does attention visualization in the proposed model reflect more dynamic and context-aware attention compared to standard Transformers?
• RQ5 Can label smoothing and architectural modifications improve generalization on unseen test data?

Key findings

• The proposed Transformer model achieved a private RMSE of 0.0577 and public RMSE of 0.0588, outperforming all baseline models including DAVE2, ResNet50, and CNN-LSTM.
• The model with optical flow and speed prediction branches reduced error on extreme steering angles compared to standard models.
• Ablation studies confirmed that both the optical flow branch and speed prediction head contribute significantly to performance, with the full model outperforming a simple Transformer by 0.0079 RMSE on private data.
• Attention maps showed that the proposed model dynamically shifts focus across frames, especially on motion boundaries, unlike the static attention of standard Transformers.
• The model ranked 3rd or 4th in the Udacity challenge, surpassing transfer learning and LSTM-based approaches but falling short of the first-place solution (0.0483 RMSE).
• Smoothing the predictions with a factor of 0.35 further improved performance, indicating improved generalization and robustness to noisy labels.