[Paper Review] End-to-End Deep Reinforcement Learning for Lane Keeping Assist

Ahmad El Sallab, Mohammed Abdou+2 more|arXiv (Cornell University)|Dec 13, 2016

Reinforcement Learning in RoboticsComputer Science21 references142 citations

TL;DR

The paper explores end-to-end deep reinforcement learning for lane keeping using discrete (DQN) and continuous (DDAC) action spaces in TORCS, comparing performance and the impact of termination constraints on learning convergence.

ABSTRACT

Reinforcement learning is considered to be a strong AI paradigm which can be used to teach machines through interaction with the environment and learning from their mistakes, but it has not yet been successfully used for automotive applications. There has recently been a revival of interest in the topic, however, driven by the ability of deep learning algorithms to learn good representations of the environment. Motivated by Google DeepMind's successful demonstrations of learning for games from Breakout to Go, we will propose different methods for autonomous driving using deep reinforcement learning. This is of particular interest as it is difficult to pose autonomous driving as a supervised learning problem as it has a strong interaction with the environment including other vehicles, pedestrians and roadworks. As this is a relatively new area of research for autonomous driving, we will formulate two main categories of algorithms: 1) Discrete actions category, and 2) Continuous actions category. For the discrete actions category, we will deal with Deep Q-Network Algorithm (DQN) while for the continuous actions category, we will deal with Deep Deterministic Actor Critic Algorithm (DDAC). In addition to that, We will also discover the performance of these two categories on an open source car simulator for Racing called (TORCS) which stands for The Open Racing car Simulator. Our simulation results demonstrate learning of autonomous maneuvering in a scenario of complex road curvatures and simple interaction with other vehicles. Finally, we explain the effect of some restricted conditions, put on the car during the learning phase, on the convergence time for finishing its learning phase.

Motivation & Objective

• Motivate the use of reinforcement learning for autonomous driving due to interactive driving environments.
• Investigate end-to-end models that map raw sensor inputs to driving actions without hand-crafted features.
• Compare discrete-action (DQN) and continuous-action (DDAC) DRL approaches for lane keeping.
• Assess how restricted termination conditions affect learning convergence time.

Proposed method

• Formulate lane keeping as a DRL problem with sensor fusion of camera, LIDAR, and radar inputs.
• Apply two DRL paradigms: Deep Q-Network (DQN) for discrete actions and Deep Deterministic Actor-Critic (DDAC) for continuous actions.
• Train end-to-end networks on the TORCS simulator using trackPos and car speed as inputs and steering, gear, acceleration, and brake as outputs.
• Discretize actions via tile coding for DQN and use policy gradient with actor-critic for DDAC.
• Evaluate performance on straight and curved track sections to compare convergence and trajectory quality.
• Examine the effect of termination conditions (No termination, Out of Track, Stuck, Out of Track with Stuck) on convergence time.

Experimental results

Research questions

• RQ1 Can end-to-end DRL models learn lane keeping from raw sensor inputs without hand-crafted features?
• RQ2 How do discrete (DQN) and continuous (DDAC) action formulations compare in terms of learning efficacy and trajectory smoothness?
• RQ3 What is the impact of different termination conditions on learning convergence time for DRL-based lane keeping?
• RQ4 Does DDAC provide smoother control and better performance on curved track sections compared to DQN?

Key findings

• DDAC yields smoother steering and better performance on curved track parts compared to DQN with tiled discrete actions.
• DDQN (DQN with tile coding) converges faster in some setups but may produce more abrupt steering actions.
• No termination conditions lead to faster convergence than setups with restricted termination, but risk poorer exploration and local minima.
• Restricting termination conditions generally increases convergence time due to more frequent episode resets.
• On straight track sections, both methods perform similarly, while on curved sections DDAC outperforms DQN.