[Paper Review] Beyond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars

Mark Martinez, Chawin Sitawarin+4 more|arXiv (Cornell University)|Dec 4, 2017

Autonomous Vehicle Technology and Safety14 references59 citations

TL;DR

The paper uses GTA-V to collect labeled affordance data to train a CNN for highway driving variables, and introduces Unity-based Princeton Virtual Environment (PVE) for corner-case testing and model enhancement; it reports training results and discusses limitations and future directions.

ABSTRACT

As an initial assessment, over 480,000 labeled virtual images of normal highway driving were readily generated in Grand Theft Auto V's virtual environment. Using these images, a CNN was trained to detect following distance to cars/objects ahead, lane markings, and driving angle (angular heading relative to lane centerline): all variables necessary for basic autonomous driving. Encouraging results were obtained when tested on over 50,000 labeled virtual images from substantially different GTA-V driving environments. This initial assessment begins to define both the range and scope of the labeled images needed for training as well as the range and scope of labeled images needed for testing the definition of boundaries and limitations of trained networks. It is the efficacy and flexibility of a "GTA-V"-like virtual environment that is expected to provide an efficient well-defined foundation for the training and testing of Convolutional Neural Networks for safe driving. Additionally, described is the Princeton Virtual Environment (PVE) for the training, testing and enhancement of safe driving AI, which is being developed using the video-game engine Unity. PVE is being developed to recreate rare but critical corner cases that can be used in re-training and enhancing machine learning models and understanding the limitations of current self driving models. The Florida Tesla crash is being used as an initial reference.

Motivation & Objective

• Demonstrate that GTA-V can yield large, labeled datasets to train CNNs for driving-affordance estimation.
• Identify a set of 8 affordance variables usable by direct perception for highway driving.
• Show initial CNN training results and analyze boundaries/limitations of virtual data for autonomous driving models.
• Propose a Unity-based virtual environment (PVE) to recreate rare corner cases for training and testing.

Proposed method

• Adapt AlexNet-inspired CNN to perform regression for 8 continuous affordance outputs (angle, car_L, car_M, car_R, lane_LL, lane_L, lane_R, lane_RR).
• Modify network to output 8 real-valued neurons with tanh activation and mean-squared error loss.
• Collect ~483,000 GTA-V images with ground-truth affordances via Rage Plugin Hook; use 350,000 for training and 133,000 for testing/validation.
• Preprocess inputs by mean subtraction and rescale active/inactive affordances to [-0.9,0.9] to reflect the network output range, treating inactive affordances as out-of-range.
• Train over 21 epochs (≈36 hours) on a Tesla K40 with batch size 32; report per-affordance validation losses.
• Describe a Unity-based pipeline (Mapzen-derived data) to generate controllable synthetic driving scenarios and corner cases (e.g., pedestrian crossing, Tesla Florida crash) for retraining.

Experimental results

Research questions

• RQ1 Can GTA-V be used to generate labeled data sufficient for training CNN-based direct-perception models for highway driving?
• RQ2 What are the 8 affordance variables useful for direct perception in autonomous driving from image data?
• RQ3 How do CNN-estimated affordances (angle and distances) compare to ground-truth in GTA-V test data, and what are the main sources of error?
• RQ4 Can a Unity-based environment (PVE) recreate useful corner cases to enhance training and understanding of model limitations?
• RQ5 What are practical limitations and future directions for bridging virtual data with real-world driving systems?

Key findings

• The CNN achieved final training loss per sample per variable of 0.151 and validation loss per sample per variable of 0.314 across 8 affordances.
• Angle to road and lane-distance estimates yielded the lowest validation losses, while car-distance estimates were higher and showed signs of overfitting for lane-related variables.
• Lane distance estimates were more precise than car distance estimates; errors in car_M were larger when cars were in adjacent left/right lanes or at greater distance.
• Training on GTA-V with ≈483,000 images demonstrated the feasibility of using virtual data for training and testing autonomous driving affordances.
• A Unity-based environment (PVE) enables controlled creation of corner cases (e.g., pedestrian crossing, Tesla crash scenarios) not readily available in GTA-V, supporting retraining and enhancement of models.
• Authors acknowledge GTA-V limitations (legal, extensibility, realism) and propose future work combining real and virtual data and exploring time-dependent models (e.g., 3D CNNs, LSTMs).