[Paper Review] Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving

Shai Shalev‐Shwartz, Shaked Shammah+1 more|arXiv (Cornell University)|Oct 11, 2016

Reinforcement Learning in Robotics31 references367 citations

TL;DR

The paper introduces a safe reinforcement learning framework for autonomous driving that separates learning desires from hard-constrained trajectory planning and uses an Option Graph for hierarchical temporal abstraction to reduce variance and sample complexity, demonstrated on a challenging double-merge scenario.

ABSTRACT

Autonomous driving is a multi-agent setting where the host vehicle must apply sophisticated negotiation skills with other road users when overtaking, giving way, merging, taking left and right turns and while pushing ahead in unstructured urban roadways. Since there are many possible scenarios, manually tackling all possible cases will likely yield a too simplistic policy. Moreover, one must balance between unexpected behavior of other drivers/pedestrians and at the same time not to be too defensive so that normal traffic flow is maintained. In this paper we apply deep reinforcement learning to the problem of forming long term driving strategies. We note that there are two major challenges that make autonomous driving different from other robotic tasks. First, is the necessity for ensuring functional safety - something that machine learning has difficulty with given that performance is optimized at the level of an expectation over many instances. Second, the Markov Decision Process model often used in robotics is problematic in our case because of unpredictable behavior of other agents in this multi-agent scenario. We make three contributions in our work. First, we show how policy gradient iterations can be used without Markovian assumptions. Second, we decompose the problem into a composition of a Policy for Desires (which is to be learned) and trajectory planning with hard constraints (which is not learned). The goal of Desires is to enable comfort of driving, while hard constraints guarantees the safety of driving. Third, we introduce a hierarchical temporal abstraction we call an "Option Graph" with a gating mechanism that significantly reduces the effective horizon and thereby reducing the variance of the gradient estimation even further.

Motivation & Objective

• Address functional safety in learning-based driving policies within multi-agent traffic.
• Handle non-Markovian and multi-agent dynamics by avoiding reliance on strict MDP assumptions.
• Develop a learning framework that yields comfortable driving while guaranteeing safety via hard constraints.
• Introduce hierarchical temporal abstraction to reduce gradient variance and sample complexity.

Proposed method

• Decompose policy into a learnable Policy for Desires and a non-learned trajectory planner with hard safety constraints.
• Use policy gradient methods that do not require Markov assumptions, with variance reduction techniques.
• Introduce an Option Graph to provide temporal abstraction and gating, reducing horizon and variance.
• Parameterize the Desires as a product space [0, v_max] × L × {g,t,o}^n to capture speed, lane position, and interactions.
• Translate Desires into a trajectory cost function with hard constraints to guarantee safety.

Experimental results

Research questions

• RQ1 Can policy gradient reinforcement learning operate effectively without Markovian assumptions in a driving multi-agent setting?
• RQ2 How can we ensure functional safety in RL for autonomous driving without sacrificing learning efficiency?
• RQ3 Does hierarchical temporal abstraction via an Option Graph reduce gradient variance and improve sample efficiency in driving policies?
• RQ4 Can a Desires-to-trajectory decomposition enable safe, comfortable driving under complex merge scenarios?

Key findings

• Policy gradient can be formulated without Markov assumptions for autonomous driving; unbiased gradient estimates remain feasible.
• Safety is achieved by decomposing policy into Desires (learned) and a deterministic, constraint-driven trajectory planner.
• An Option Graph provides hierarchical decisions that reduce effective horizon and gradient variance, improving sample efficiency.
• The Desires-to-trajectory framework enables handling of challenging maneuvers like double merges with functional safety guarantees.