[论文解读] A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle

Michael Teti, William Edward Hahn+3 more|arXiv (Cornell University)|Mar 26, 2018

Advanced Neural Network Applications参考文献 43被引用 3

一句话总结

本研究首次系统比较了七种深度学习架构——从全连接网络到ResNet和LSTM——在使用低成本、室内遥控车辆平台的端到端自动驾驶任务中的表现。当使用单帧彩色图像进行训练时，AlexNet表现最佳（第一阶段成功率为100%，第二阶段为55%），而VGG-16在各类输入类型中表现出最一致的性能，凸显了验证准确率与实际驾驶性能之间存在显著的“部署差距”。

ABSTRACT

Self-driving technology is advancing rapidly --- albeit with significant challenges and limitations. This progress is largely due to recent developments in deep learning algorithms. To date, however, there has been no systematic comparison of how different deep learning architectures perform at such tasks, or an attempt to determine a correlation between classification performance and performance in an actual vehicle, a potentially critical factor in developing self-driving systems. Here, we introduce the first controlled comparison of multiple deep-learning architectures in an end-to-end autonomous driving task across multiple testing conditions. We compared performance, under identical driving conditions, across seven architectures including a fully-connected network, a simple 2 layer CNN, AlexNet, VGG-16, Inception-V3, ResNet, and an LSTM by assessing the number of laps each model was able to successfully complete without crashing while traversing an indoor racetrack. We compared performance across models when the conditions exactly matched those in training as well as when the local environment and track were configured differently and objects that were not included in the training dataset were placed on the track in various positions. In addition, we considered performance using several different data types for training and testing including single grayscale and color frames, and multiple grayscale frames stacked together in sequence. With the exception of a fully-connected network, all models performed reasonably well (around or above 80\%) and most very well (~95\%) on at least one input type but with considerable variation across models and inputs. Overall, AlexNet, operating on single color frames as input, achieved the best level of performance (100\% success rate in phase one and 55\% in phase two) while VGG-16 performed well most consistently across image types.

研究动机与目标

• 在真实世界的自动驾驶任务中，对多种深度学习架构进行受控的、系统性的比较。
• 在相同条件下，评估不同输入类型（单帧、帧堆栈、灰度图、彩色图）下的模型性能。
• 研究嵌入式系统中模型验证性能与实际驾驶性能之间的差异。
• 识别影响端到端自动驾驶控制中真实世界泛化能力的网络结构与训练因素。
• 探索检测和缓解自动驾驶系统中“部署差距”的方法。

提出的方法

• 使用低成本、市售的遥控车辆，配备GPU加速计算机，实现实时推理。
• 在传感器输入与人工转向指令配对的数据上，训练七种深度学习模型——全连接网络、两层CNN、AlexNet、VGG-16、Inception-V3、ResNet-26和LSTM。
• 在两个测试阶段评估模型：(1) 训练与测试条件完全相同；(2) 面对未见过的障碍物和环境变化。
• 采用多种输入模态：单帧灰度图与彩色图，以及灰度帧的序列堆叠。
• 应用一种新颖的像素翻转方法，生成显著性热力图，识别最影响模型决策的图像特征。
• 分析不同模型之间的驾驶路径一致性和相似性，以评估行为泛化能力和偏差。

实验结果

研究问题

• RQ1 在受控的、相同条件下，不同深度学习架构在真实世界自动驾驶中的表现如何？
• RQ2 验证性能在多大程度上与嵌入式控制系统中的实际驾驶成功率相关？
• RQ3 输入类型的变化（灰度图与彩色图，单帧与帧堆栈）在多大程度上影响模型性能与泛化能力？
• RQ4 哪些网络结构或训练特征有助于实现一致的驾驶行为，并增强对未见障碍物的鲁棒性？
• RQ5 显著性图与路径相似性度量能否帮助识别易受“部署差距”影响的模型？

主要发现

• 当使用单帧彩色图像输入时，AlexNet表现最佳，第一阶段完成100%的圈数，第二阶段完成55%。
• VGG-16在所有输入类型中表现出最一致的性能，在多变的测试条件下优于大多数模型。
• 观察到显著的“部署差距”：验证性能强的模型（如ResNet-26）在实际驾驶中常因泛化能力差和行为不一致而失败。
• ResNet-26表现出高路径变异性与强烈的行为偏差，导致在面对未见障碍物或环境变化时适应能力差。
• LSTM模型在灰度图输入上表现优于彩色图输入，因其计算瓶颈导致在彩色帧上每秒仅能处理18次推理，低于所需的30 FPS。
• 模型之间的路径相似性与成功相似性之间无强相关性，表明行为一致性并非驾驶成功可靠的预测指标。