[论文解读] Beyond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars

Mark Martinez, Chawin Sitawarin+4 more|arXiv (Cornell University)|Dec 4, 2017

Autonomous Vehicle Technology and Safety参考文献 14被引用 59

一句话总结

该论文使用 GTA-V 收集带标签的可感知数据，以训练用于高速公路驾驶变量的 CNN，并引入基于 Unity 的 Princeton Virtual Environment (PVE) 进行极端场景测试和模型增强；它报告了训练结果并讨论了局限性与未来方向。

ABSTRACT

As an initial assessment, over 480,000 labeled virtual images of normal highway driving were readily generated in Grand Theft Auto V's virtual environment. Using these images, a CNN was trained to detect following distance to cars/objects ahead, lane markings, and driving angle (angular heading relative to lane centerline): all variables necessary for basic autonomous driving. Encouraging results were obtained when tested on over 50,000 labeled virtual images from substantially different GTA-V driving environments. This initial assessment begins to define both the range and scope of the labeled images needed for training as well as the range and scope of labeled images needed for testing the definition of boundaries and limitations of trained networks. It is the efficacy and flexibility of a "GTA-V"-like virtual environment that is expected to provide an efficient well-defined foundation for the training and testing of Convolutional Neural Networks for safe driving. Additionally, described is the Princeton Virtual Environment (PVE) for the training, testing and enhancement of safe driving AI, which is being developed using the video-game engine Unity. PVE is being developed to recreate rare but critical corner cases that can be used in re-training and enhancing machine learning models and understanding the limitations of current self driving models. The Florida Tesla crash is being used as an initial reference.

研究动机与目标

• 证明 GTA-V 能产出大规模带标签的数据集，用于训练用于驾驶可感知估计的 CNN。
• 识别一组可供直接感知用于高速公路驾驶的 8 个可感知变量。
• 展示初步的 CNN 训练结果，并分析虚拟数据在自动驾驶模型中的边界与局限。
• 提出一个基于 Unity 的虚拟环境（PVE），用于重现稀有的极端场景以用于训练与测试。

提出的方法

• 改编受 AlexNet 启发的 CNN，使其执行 8 个连续可感知变量的回归输出（角度、car_L、car_M、car_R、lane_LL、lane_L、lane_R、lane_RR）。
• 将网络修改为输出 8 个实值神经元，采用 tanh 激活和均方误差损失。
• 通过 Rage Plugin Hook 收集约 483,000 张带有真实标签的 GTA-V 图像；其中 350,000 用于训练，133,000 用于测试/验证。
• 通过均值子减和将活动/非活动的可感知变量重新缩放至 [-0.9,0.9]，以反映网络输出范围，将非活动的可感知变量视为超出范围。
• 在 Tesla K40 上以 batch size 32 训练 21 个 epoch（约 36 小时）；报告每个可感知变量的验证损失。
• 描述一个基于 Unity 的管线（基于 Mapzen 的数据），用于生成可控的合成驾驶场景和极端案例（如行人穿越、特斯拉佛州事故）以用于再训练。

实验结果

研究问题

• RQ1 Can GTA-V be used to generate labeled data sufficient for training CNN-based direct-perception models for highway driving?
• RQ2 What are the 8 affordance variables useful for direct perception in autonomous driving from image data?
• RQ3 How do CNN-estimated affordances (angle and distances) compare to ground-truth in GTA-V test data, and what are the main sources of error?
• RQ4 Can a Unity-based environment (PVE) recreate useful corner cases to enhance training and understanding of model limitations?
• RQ5 What are practical limitations and future directions for bridging virtual data with real-world driving systems?

主要发现

• The CNN achieved final training loss per sample per variable of 0.151 and validation loss per sample per variable of 0.314 across 8 affordances.
• Angle to road and lane-distance estimates yielded the lowest validation losses, while car-distance estimates were higher and showed signs of overfitting for lane-related variables.
• Lane distance estimates were more precise than car distance estimates; errors in car_M were larger when cars were in adjacent left/right lanes or at greater distance.
• Training on GTA-V with ≈483,000 images demonstrated the feasibility of using virtual data for training and testing autonomous driving affordances.
• A Unity-based environment (PVE) enables controlled creation of corner cases (e.g., pedestrian crossing, Tesla crash scenarios) not readily available in GTA-V, supporting retraining and enhancement of models.
• Authors acknowledge GTA-V limitations (legal, extensibility, realism) and propose future work combining real and virtual data and exploring time-dependent models (e.g., 3D CNNs, LSTMs).