[论文解读] Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning

Meixin Zhu, Xuesong Wang+1 more|arXiv (Cornell University)|Jan 3, 2019

Traffic control and management参考文献 39被引用 27

一句话总结

本文提出了一种基于深度强化学习（DRL）的人类驾驶风格自主跟驰模型，其中智能体通过基于速度和间距差异的奖励函数，从真实驾驶数据中学习。DDPGvRT 模型实现了卓越的准确性——间距误差为18%，速度误差为5%，优于传统模型和数据驱动模型，同时在不同驾驶场景中具备良好的泛化能力，并通过持续学习适应不同驾驶员。

ABSTRACT

This study proposes a framework for human-like autonomous car-following planning based on deep reinforcement learning (deep RL). Historical driving data are fed into a simulation environment where an RL agent learns from trial and error interactions based on a reward function that signals how much the agent deviates from the empirical data. Through these interactions, an optimal policy, or car-following model that maps in a human-like way from speed, relative speed between a lead and following vehicle, and inter-vehicle spacing to acceleration of a following vehicle is finally obtained. The model can be continuously updated when more data are fed in. Two thousand car-following periods extracted from the 2015 Shanghai Naturalistic Driving Study were used to train the model and compare its performance with that of traditional and recent data-driven car-following models. As shown by this study results, a deep deterministic policy gradient car-following model that uses disparity between simulated and observed speed as the reward function and considers a reaction delay of 1s, denoted as DDPGvRT, can reproduce human-like car-following behavior with higher accuracy than traditional and recent data-driven car-following models. Specifically, the DDPGvRT model has a spacing validation error of 18% and speed validation error of 5%, which are less than those of other models, including the intelligent driver model, models based on locally weighted regression, and conventional neural network-based models. Moreover, the DDPGvRT demonstrates good capability of generalization to various driving situations and can adapt to different drivers by continuously learning. This study demonstrates that reinforcement learning methodology can offer insight into driver behavior and can contribute to the development of human-like autonomous driving algorithms and traffic-flow models.

研究动机与目标

• 开发一种基于深度强化学习的人类驾驶风格自主跟驰模型，以复现真实驾驶员行为。
• 通过从自然驾驶数据中学习，改进传统和近期的数据驱动跟驰模型。
• 通过增量学习实现模型对不同驾驶员和驾驶条件的持续适应。
• 利用真实世界驾驶数据，将模型性能与既定基准进行验证。
• 探索强化学习在智能交通系统中对复杂驾驶员行为建模的潜力。

提出的方法

• 采用深度确定性策略梯度（DDPG）算法，训练智能体将车辆状态（速度、相对速度、车头间距）映射为加速度动作。
• 奖励函数定义为模拟与观测车辆速度之间的负差异，以鼓励智能体模仿真实人类驾驶模式。
• 在环境中显式建模了1秒的反应延迟，以反映真实的人类反应时间。
• 训练环境基于2015年上海自然驾驶研究中的2,000个跟驰时段构建。
• 随着新数据的输入，模型持续更新，实现对新驾驶行为的在线适应。
• 通过与基线模型对比的间距和速度验证误差评估性能。

实验结果

研究问题

• RQ1 深度强化学习能否有效从真实世界驾驶数据中学习人类驾驶风格的跟驰行为？
• RQ2 与传统模型（如智能驾驶员模型）和基于回归或神经网络的数据驱动模型相比，DDPGvRT 模型在准确性上表现如何？
• RQ3 DDPGvRT 模型在多样化驾驶场景中的泛化能力如何？其对不同驾驶员的适应能力如何？
• RQ4 在训练环境中引入1秒的反应延迟，对所学策略的真实性和性能有何影响？
• RQ5 模型能否通过新数据的增量更新，长期保持相关性和准确性？

主要发现

• DDPGvRT 模型的间距验证误差为18%，显著低于其他模型，表明其在复现人类间距行为方面具有高保真度。
• 该模型的速度验证误差为5%，表明其在模仿真实驾驶员速度调节方面具有卓越准确性。
• 与智能驾驶员模型和局部加权回归模型相比，DDPGvRT 在所有评估指标上均表现出一致的性能提升。
• 该模型能良好泛化至未见过的驾驶场景，在无需重新训练的情况下保持稳定性能。
• 通过持续学习，该模型能有效适应不同驾驶员，表明其具备强大的个性化潜力。
• 在训练环境中整合1秒的反应延迟，显著提升了所学策略的真实性和性能。