[Paper Review] Self-Driving Car Steering Angle Prediction Based on Image Recognition

Shuyang Du, Haoli Guo+1 more|arXiv (Cornell University)|Dec 11, 2019

Autonomous Vehicle Technology and Safety20 references83 citations

TL;DR

The paper compares a 3D CNN+LSTM architecture and a ResNet50-based transfer learning model for predicting steering angles from Udacity’s self-driving car images, reporting RMSE results and model insights.

ABSTRACT

Self-driving vehicles have expanded dramatically over the last few years. Udacity has release a dataset containing, among other data, a set of images with the steering angle captured during driving. The Udacity challenge aimed to predict steering angle based on only the provided images. We explore two different models to perform high quality prediction of steering angles based on images using different deep learning techniques including Transfer Learning, 3D CNN, LSTM and ResNet. If the Udacity challenge was still ongoing, both of our models would have placed in the top ten of all entries.

Motivation & Objective

• Motivate end-to-end steering angle prediction from driving images to reduce hand-crafted rules.
• Explore temporal information through 3D convolutions and LSTM for steering angle prediction.
• Evaluate transfer learning with a pretrained model (ResNet50) on the Udacity dataset.
• Assess data augmentation strategies and their impact on model performance.
• Benchmark against NVIDIA’s baseline and discuss implications for production systems.

Proposed method

• Develop a 3D convolutional model with residual connections followed by LSTM layers to capture temporal information (5 sequences of 5 frames, 5x5x120x320x3, 543,131 parameters).
• Develop a transfer learning model using ResNet50 pretrained on ImageNet; freeze first ~45 layers and attach fully connected layers to predict steering angle (1 output).
• Apply data augmentation including brightness, shadow, shifts, and rotations; normalize inputs to [-1,1]; resize crops for compatibility with networks.
• Train using mean squared error loss and Adam optimizer with learning rate decay; report RMSE as evaluation metric.
• Visualize saliency maps to interpret model focus and discuss temporal feature utilization in the 3D-LSTM model.

Experimental results

Research questions

• RQ1 Can a 3D CNN + LSTM architecture leverage temporal information to predict steering angles from image sequences?
• RQ2 Does transfer learning with a pretrained ResNet50 achieve competitive steering angle RMSE on the Udacity dataset?
• RQ3 How do different data augmentation levels affect model performance and generalization?
• RQ4 How do the proposed models compare to the NVIDIA baseline and to top entries in the Udacity challenge?

Key findings

• The 3D CNN + LSTM model achieves RMSE of 0.1123 on the test set (placing about 10th on the leaderboard).
• The ResNet50 transfer learning model achieves the best RMSE of 0.0709 on the test set (placing about 4th on the leaderboard).
• Minimal data augmentation outperformed heavier augmentations for the tested epochs; moderate augmentation performed reasonably but sometimes lagged behind minimal.
• Saliency maps indicate the models focus on lane markers and nearby objects; temporal saliency in the 3D-LSTM model highlights changing features frame-to-frame.
• The NVIDIA baseline achieves RMSEs of 0.0986 (test) and 0.0750 (training) under the same evaluation setup, illustrating competitive performance of the proposed methods.
• Overall, transfer learning with ResNet50 provides strong performance without explicit temporal modeling, while the 3D-LSTM model demonstrates the value of incorporating temporal context under computational constraints.