[Paper Review] Towards End-to-End Deep Learning for Autonomous Racing: On Data Collection and a Unified Architecture for Steering and Throttle Prediction

Shakti N. Wadekar, Benjamin J. Schwartz+6 more|arXiv (Cornell University)|May 4, 2021

Autonomous Vehicle Technology and Safety17 references11 citations

TL;DR

This paper proposes an end-to-end deep learning framework for autonomous racing that jointly predicts steering and throttle using a unified convolutional neural network architecture without recurrent connections or feedback. It demonstrates that freezing pre-trained steering convolutional features and fine-tuning only the final fully connected layers for throttle prediction enables stable high-speed control, achieving 90 mph on a track with sharp turns using only imitation learning and data augmentation.

ABSTRACT

Deep Neural Networks (DNNs) which are trained end-to-end have been successfully applied to solve complex problems that we have not been able to solve in past decades. Autonomous driving is one of the most complex problems which is yet to be completely solved and autonomous racing adds more complexity and exciting challenges to this problem. Towards the challenge of applying end-to-end learning to autonomous racing, this paper shows results on two aspects: (1) Analyzing the relationship between the driving data used for training and the maximum speed at which the DNN can be successfully applied for predicting steering angle, (2) Neural network architecture and training methodology for learning steering and throttle without any feedback or recurrent connections.

Motivation & Objective

• To investigate data collection strategies for end-to-end deep learning in high-speed autonomous racing, focusing on the relationship between training data quantity and maximum achievable speed.
• To develop a unified deep neural network architecture that jointly predicts steering and throttle without using recurrent units (e.g., LSTMs) or feedback loops.
• To enable efficient joint learning by reusing convolutional features trained for steering during throttle prediction, minimizing retraining and computational cost.
• To validate the proposed method on diverse track layouts, including high-curvature tracks and standard simulator tracks, under real-time inference conditions.
• To establish a data-centric foundation for future scalable deployment of end-to-end models in autonomous racing systems.

Proposed method

• Utilized a custom Unity-based simulator with three front-facing cameras, collecting 10 Hz data for images, steering, throttle, and speed during training laps.
• Applied Direct Policy Learning (a form of imitation learning combining behavioral cloning and data aggregation) with expert feedback to iteratively improve policy performance.
• Trained a base convolutional neural network (inspired by NVIDIA’s end-to-end self-driving architecture) for steering prediction using a fixed, simplified architecture for low-latency inference.
• For throttle prediction, froze the convolutional layers from the trained steering model and only fine-tuned the subsequent fully connected layers, enabling feature transfer without retraining the backbone.
• Merged the two models into a single unified architecture where shared convolutional layers serve as a common backbone, with separate heads for steering and throttle prediction.
• Used data augmentation and iterative expert correction to improve generalization and robustness across different track geometries and speed regimes.

Experimental results

Research questions

• RQ1 How does the amount of collected training data (measured in laps) correlate with the maximum stable speed achievable in end-to-end deep learning for autonomous racing?
• RQ2 Can a unified deep neural network architecture predict both steering and throttle without using recurrent connections or feedback mechanisms?
• RQ3 To what extent can pre-trained convolutional features from a steering model be effectively transferred to a throttle prediction task with minimal additional training?
• RQ4 How does the model perform on tracks with high-curvature turns compared to standard or straight tracks under high-speed conditions?
• RQ5 What data collection strategies are most effective in enabling stable, high-speed autonomous control in simulated racing environments?

Key findings

• On Track 2 with sharp turns, the model successfully achieved a maximum speed of 90 mph by reducing throttle during turns and accelerating on straights, demonstrating high-speed stability.
• The model maintained stable performance across multiple laps on both Track 2 and the default Lake Track, indicating generalization across different track geometries.
• The relationship between training data quantity (number of laps) and maximum achievable speed was empirically validated, showing that increased data improves high-speed capability.
• Freezing the convolutional layers from the steering model and fine-tuning only the fully connected layers for throttle prediction yielded high performance with minimal additional training time.
• The unified architecture achieved joint steering and throttle prediction without recurrence or feedback, proving that such components are not necessary for high-speed control in this setting.
• Video demonstrations confirmed the model’s ability to navigate complex tracks autonomously, with real-time inference and smooth control transitions.