[Paper Review] DeepTraffic: Driving Fast through Dense Traffic with Deep Reinforcement Learning.

Lex Fridman, Benedikt Jenik+1 more|arXiv (Cornell University)|Jan 9, 2018

Traffic control and management19 citations

TL;DR

DeepTraffic introduces a micro-traffic simulation enabling end-to-end deep reinforcement learning for autonomous driving, where a single neural network handles perception, control, and planning. The system leverages a large-scale, open competition to crowd-source hyperparameter tuning, resulting in significant performance gains through collective hyperparameter optimization across thousands of submissions.

ABSTRACT

We present a micro-traffic simulation (named DeepTraffic) where the perception, control, and planning systems for one of the cars are all handled by a single neural network as part of a model-free, off-policy reinforcement learning process. The primary goal of DeepTraffic is to make the hands-on study of deep reinforcement learning accessible to thousands of students, educators, and researchers in order to inspire and fuel the exploration and evaluation of DQN variants and hyperparameter configurations through large-scale, open competition. This paper investigates the crowd-sourced hyperparameter tuning of the policy network that resulted from the first iteration of the DeepTraffic competition where thousands of participants actively searched through the hyperparameter space with the objective of their neural network submission to make it onto the top-10 leaderboard.

Motivation & Objective

• To create an accessible, scalable micro-traffic simulation for studying deep reinforcement learning in autonomous driving.
• To enable thousands of students, educators, and researchers to experiment with DQN variants and hyperparameter configurations.
• To investigate the effectiveness of crowd-sourced hyperparameter tuning in improving deep reinforcement learning policy performance.
• To inspire large-scale, open competitions for evaluating and advancing deep reinforcement learning methods.

Proposed method

• A model-free, off-policy deep reinforcement learning framework is implemented in a custom micro-traffic simulation environment (DeepTraffic).
• A single deep neural network processes raw observations to directly output actions, enabling end-to-end learning.
• The system supports training and evaluation of DQN-based agents in a dense traffic environment with realistic dynamics.
• Hyperparameter tuning is conducted via a large-scale, open competition involving thousands of participants.
• The competition enables systematic exploration of hyperparameter space across diverse configurations.
• Performance is evaluated using a leaderboard ranking top-performing agents based on cumulative reward and success metrics.

Experimental results

Research questions

• RQ1 How effective is crowd-sourced hyperparameter tuning in improving deep reinforcement learning policy performance in a complex traffic environment?
• RQ2 What hyperparameter configurations yield the highest performance in end-to-end deep reinforcement learning for autonomous driving?
• RQ3 How does the collective exploration of hyperparameter space compare to individual or heuristic tuning approaches?
• RQ4 What insights can be gained from analyzing the distribution of successful hyperparameters across thousands of submissions?

Key findings

• The top-performing agents achieved significantly higher cumulative rewards compared to baseline DQN configurations, demonstrating the value of large-scale hyperparameter search.
• A diverse set of hyperparameter configurations led to high performance, indicating multiple viable paths to success in the task.
• The competition revealed that certain hyperparameter combinations—particularly learning rate and experience replay buffer size—had a strong impact on final performance.
• The collective effort of thousands of participants uncovered non-obvious, high-performing configurations that outperformed standard DQN settings.