[Paper Review] Deep Reinforcement Learning for Human-Like Driving Policies in Collision Avoidance Tasks of Self-Driving Cars

Ran Emuna, Avinoam Borowsky+1 more|arXiv (Cornell University)|Jun 7, 2020

Autonomous Vehicle Technology and Safety42 references21 citations

TL;DR

This paper proposes a model-free deep reinforcement learning (DRL) approach to imitate expert human driving behavior in static obstacle avoidance tasks on a two-lane highway. By combining rule-based driving constraints with data-driven expert demonstrations through a stochastic reward function, the method generates human-like driving policies that closely match the mean and variance of human track position and speed distributions, achieving near-human performance in a visual Turing test.

ABSTRACT

The technological and scientific challenges involved in the development of autonomous vehicles (AVs) are currently of primary interest for many automobile companies and research labs. However, human-controlled vehicles are likely to remain on the roads for several decades to come and may share with AVs the traffic environments of the future. In such mixed environments, AVs should deploy human-like driving policies and negotiation skills to enable smooth traffic flow. To generate automated human-like driving policies, we introduce a model-free, deep reinforcement learning approach to imitate an experienced human driver's behavior. We study a static obstacle avoidance task on a two-lane highway road in simulation (Unity). Our control algorithm receives a stochastic feedback signal from two sources: a model-driven part, encoding simple driving rules, such as lane-keeping and speed control, and a stochastic, data-driven part, incorporating human expert knowledge from driving data. To assess the similarity between machine and human driving, we model distributions of track position and speed as Gaussian processes. We demonstrate that our approach leads to human-like driving policies.

Motivation & Objective

• To develop a self-driving car policy that mimics experienced human drivers in mixed-traffic environments.
• To integrate rule-based driving constraints (e.g., lane-keeping, speed control) with data-driven expert knowledge from real driving data.
• To evaluate the similarity between machine and human driving using statistical distributions of kinematic features.
• To test the generalization capability of the learned policy on new road configurations with unseen obstacle distributions.
• To assess whether the machine-driven agent can pass a visual Turing test by appearing human-like to observers.

Proposed method

• A deep reinforcement learning agent is trained using Proximal Policy Optimization (PPO) with a hybrid reward function combining model-driven rules and data-driven expert behavior.
• The reward function incorporates stochastic feedback from two sources: rule-based components (e.g., lane-keeping, speed control) and human expert data via Gaussian process modeling of track position and speed.
• State distributions of human driving are modeled using Gaussian processes to capture both mean behavior and variability (first and second moments).
• The simulation environment is built in Unity, featuring a 4.7 km two-lane highway with static obstacles placed at arbitrary locations.
• A visual Turing test was conducted at the Israeli Conference on Robotics (ICR) 2019, where participants viewed recorded video clips of human and agent driving to distinguish between them.
• Generalization was evaluated on new road tracks with varying obstacle distributions, including alternating, randomized, and batched obstacles.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn to imitate the mean and variance of human driving behavior in a static obstacle avoidance task?
• RQ2 How does the inclusion of stochastic feedback from expert data affect policy generalization to new obstacle distributions?
• RQ3 To what extent can a machine-driven agent pass a visual Turing test by appearing human-like to observers?
• RQ4 How does the agent perform in novel scenarios, such as high-curvature road sections not seen during training?
• RQ5 What is the relative contribution of rule-based constraints and data-driven expert knowledge in achieving human-like driving behavior?

Key findings

• The agent successfully reproduced the first and second moments of human driving behavior, particularly in track position, with speed matching being less accurate due to partial observability.
• The stochastic reward function led to better generalization than a deterministic one, enabling the agent to avoid all obstacles on new roads with diverse obstacle distributions.
• In a visual Turing test at ICR 2019, 60% of participants failed to correctly identify the machine-driven agent, indicating strong human-like behavior.
• The agent failed to avoid obstacles in high-curvature road sections not present in the training data, highlighting a limitation in generalization to significantly novel scenarios.
• The method demonstrated that combining rule-based constraints with data-driven expert knowledge via a stochastic reward function yields effective, human-like driving policies in simulation.
• Hyperparameter tuning using PPO with specific values (e.g., learning rate 1e-1, discount γ=0.98, clipping ε=0.2) led to stable training and policy convergence.