[Paper Review] Deep Reinforcement Learning for Autonomous Driving

Wang, Sen, Daoyuan Jia+1 more|arXiv (Cornell University)|Nov 28, 2018

Reinforcement Learning in RoboticsComputer Science16 references44 citations

TL;DR

This paper applies Deep Deterministic Policy Gradient (DDPG) to autonomous driving in the TORCS simulator, designing a custom sensor input set and reward function to handle continuous action spaces and safety constraints.

ABSTRACT

Reinforcement learning has steadily improved and outperform human in lots of traditional games since the resurgence of deep neural network. However, these success is not easy to be copied to autonomous driving because the state spaces in real world are extreme complex and action spaces are continuous and fine control is required. Moreover, the autonomous driving vehicles must also keep functional safety under the complex environments. To deal with these challenges, we first adopt the deep deterministic policy gradient (DDPG) algorithm, which has the capacity to handle complex state and action spaces in continuous domain. We then choose The Open Racing Car Simulator (TORCS) as our environment to avoid physical damage. Meanwhile, we select a set of appropriate sensor information from TORCS and design our own rewarder. In order to fit DDPG algorithm to TORCS, we design our network architecture for both actor and critic inside DDPG paradigm. To demonstrate the effectiveness of our model, We evaluate on different modes in TORCS and show both quantitative and qualitative results.

Motivation & Objective

• Motivate and address the challenges of applying deep reinforcement learning to autonomous driving with continuous actions and complex states.
• Evaluate a DDPG-based agent in TORCS to learn fast, safe driving policies.
• Design sensor inputs and a tailored reward function suitable for TORCS and continuous control.
• Develop actor-critic network architectures within the DDPG framework for autonomous driving tasks.

Proposed method

• Use DDPG to learn continuous control policies for steering, acceleration, and braking.
• Select a 29-dimensional sensor input vector from TORCS as state representation.
• Define a reward function that favors speed along the track, penalizes deviation from the track center and vertical speed components.
• Design actor and critic networks with a specific architectural arrangement and an experience replay strategy.
• Incorporate target networks and soft updates to stabilize learning.

Experimental results

Research questions

• RQ1 Can DDPG learn effective continuous control policies for autonomous driving in a simulator?
• RQ2 How should sensor inputs and reward design be tailored to TORCS to facilitate learning?
• RQ3 What network architecture and stabilization techniques (e.g., target networks, replay buffers) improve learning efficiency for this task?
• RQ4 How does the agent perform across different TORCS modes (training vs. competitive) and driving scenarios?

Key findings

• The DDPG-based agent can learn to drive fast in the TORCS simulator while maintaining functional safety in the training setup.
• Training shows increasing mean speed and step-gain over episodes, stabilizing after about 100 episodes.
• The agent learns to slow down before curves to reduce drifting and improve cornering performance.
• Performance in competition mode demonstrates the agent can overtake opponents in turns and adapt to evolving scenarios.
• The training behavior includes episodes where the agent temporarily stalls or drifts, highlighting environment-induced issues that affect stability.