[Paper Review] A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle

Michael Teti, William Edward Hahn+3 more|arXiv (Cornell University)|Mar 26, 2018

Advanced Neural Network Applications43 references3 citations

TL;DR

This study presents the first systematic comparison of seven deep learning architectures—ranging from fully connected networks to ResNet and LSTM—in an end-to-end autonomous driving task using a low-cost, indoor remote-controlled vehicle platform. AlexNet achieved the highest performance (100% success rate in phase one, 55% in phase two) when trained on single color frames, while VGG-16 showed the most consistent performance across input types, highlighting a significant 'deployment gap' between validation accuracy and real-world driving performance.

ABSTRACT

Self-driving technology is advancing rapidly --- albeit with significant challenges and limitations. This progress is largely due to recent developments in deep learning algorithms. To date, however, there has been no systematic comparison of how different deep learning architectures perform at such tasks, or an attempt to determine a correlation between classification performance and performance in an actual vehicle, a potentially critical factor in developing self-driving systems. Here, we introduce the first controlled comparison of multiple deep-learning architectures in an end-to-end autonomous driving task across multiple testing conditions. We compared performance, under identical driving conditions, across seven architectures including a fully-connected network, a simple 2 layer CNN, AlexNet, VGG-16, Inception-V3, ResNet, and an LSTM by assessing the number of laps each model was able to successfully complete without crashing while traversing an indoor racetrack. We compared performance across models when the conditions exactly matched those in training as well as when the local environment and track were configured differently and objects that were not included in the training dataset were placed on the track in various positions. In addition, we considered performance using several different data types for training and testing including single grayscale and color frames, and multiple grayscale frames stacked together in sequence. With the exception of a fully-connected network, all models performed reasonably well (around or above 80\%) and most very well (~95\%) on at least one input type but with considerable variation across models and inputs. Overall, AlexNet, operating on single color frames as input, achieved the best level of performance (100\% success rate in phase one and 55\% in phase two) while VGG-16 performed well most consistently across image types.

Motivation & Objective

• To conduct a controlled, systematic comparison of multiple deep learning architectures in a real-world autonomous driving task.
• To evaluate model performance under identical conditions across different input types (single frames, frame stacks, grayscale, color).
• To investigate the discrepancy between model validation performance and actual driving performance in embedded systems.
• To identify architectural and training factors that influence real-world generalization in end-to-end autonomous control.
• To explore methods for detecting and mitigating the 'deployment gap' in self-driving systems.

Proposed method

• Used a low-cost, off-the-shelf remote-controlled vehicle equipped with a GPU-powered computer for real-time inference.
• Trained seven deep learning models—fully connected, 2-layer CNN, AlexNet, VGG-16, Inception-V3, ResNet-26, and LSTM—on sensor inputs paired with human steering commands.
• Evaluated models across two testing phases: (1) identical training and test conditions, and (2) with unseen obstacles and environmental changes.
• Employed multiple input modalities: single grayscale and color frames, and stacked sequences of grayscale frames.
• Applied a novel pixel-flipping method to generate saliency heatmaps, identifying which image features most influenced model decisions.
• Analyzed driving path consistency and similarity across models to assess behavioral generalization and bias.

Experimental results

Research questions

• RQ1 How do different deep learning architectures perform in real-world autonomous driving under controlled, identical conditions?
• RQ2 To what extent does validation performance correlate with actual driving success in embedded control systems?
• RQ3 How do variations in input type (grayscale vs. color, single frame vs. frame stack) affect model performance and generalization?
• RQ4 What architectural or training characteristics contribute to consistent driving behavior and robustness to unseen obstacles?
• RQ5 Can saliency maps and path similarity metrics help identify models prone to the 'deployment gap'?

Key findings

• AlexNet achieved the highest performance, completing 100% of laps in phase one and 55% in phase two when using single color frame inputs.
• VGG-16 demonstrated the most consistent performance across all input types, outperforming most models in diverse testing conditions.
• A significant 'deployment gap' was observed: models with strong validation performance (e.g., ResNet-26) often failed in real-world driving due to poor generalization and behavioral inconsistency.
• ResNet-26 exhibited high path variability and strong action bias, leading to poor adaptation when faced with unseen obstacles or environmental changes.
• The LSTM model performed better on grayscale inputs than color inputs due to computational bottlenecks, as it could only process 18 inferences per second on color frames, below the required 30 FPS.
• Path similarity between models did not correlate strongly with success similarity, indicating that behavioral consistency is not a reliable predictor of driving success.