[论文解读] Automated Driving Maneuvers under Interactive Environment based on Deep Reinforcement Learning

Pin Wang, Ching‐Yao Chan+1 more|arXiv (Cornell University)|Mar 25, 2018

Autonomous Vehicle Technology and Safety参考文献 12被引用 12

一句话总结

该论文提出了一种具有连续动作Q函数近似器的深度Q-learning框架，用于在交互式驾驶环境中学习安全、舒适且高效的变道操作。通过嵌入车辆动力学知识并避免使用演员-评论家结构，该方法实现了稳定的训练过程，并在仿真中表现出更优的驾驶性能，表现为奖励收敛和控制轨迹更平滑。

ABSTRACT

Safe and efficient autonomous driving maneuvers in an interactive and complex environment can be considerably challenging due to the unpredictable actions of other surrounding agents that may be cooperative or adversarial in their interactions with the ego vehicle. One of the state-of-the-art approaches is to apply Reinforcement Learning (RL) to learn a time-sequential driving policy, to execute proper control strategy or tracking trajectory in dynamic situations. However, direct application of RL algorithms is not satisfactorily enough to deal with the cases in the autonomous driving domain, mainly due to the complex driving environment and continuous action space. In this paper, we adopt Q-learning as our basic learning framework and design a unique format of the Q-function approximator that consists of neural networks to handle the continuous action space challenge. The learning model is present in a closed form of continuous control variables and trained in a simulation platform that we have developed with embedded properties of real-time vehicle interactions. The proposed algorithm avoids invoking an additional actor network that learns to take actions, as in actor-critic algorithms. At the same time, some prior knowledge of vehicle dynamics is also fed into the model to assist learning. We test our algorithm with a challenging use case - lane change maneuver, to verify the practicability and feasibility of the proposed approach. Results from accumulated rewards and vehicle performance show that RL vehicle agents successfully learn a safe, comfort and efficient driving policy as defined in the reward function.

研究动机与目标

• 开发一种强化学习框架，以处理复杂交互式驾驶环境中的连续状态和动作空间。
• 通过使用闭式Q函数近似器实现连续控制，避免演员-评论家网络的复杂性。
• 将车辆动力学的先验知识融入模型，以加速并稳定策略学习。
• 在高保真仿真环境中，通过具有挑战性的变道操作验证该方法。
• 证明智能体通过基于奖励的训练，能够学习到安全、舒适且高效的驾驶策略。

提出的方法

• 使用带归一化优势函数的二次型Q函数近似器，以在深度Q-learning中建模连续动作。
• 以闭式形式设计Q函数，消除对独立演员网络的需求，从而降低模型复杂度。
• 将车辆动力学的先验知识融入Q函数，以提高样本效率和学习稳定性。
• 在自定义仿真平台中训练模型，该平台可模拟实时车辆交互和动态交通场景。
• 采用奖励函数，对危险接近、不适感（高加速度）和低效性（长时间变道）进行惩罚。
• 使用经验回放和目标网络技术以稳定训练过程，并监控损失和奖励曲线以判断收敛性。

实验结果

研究问题

• RQ1 深度Q-learning智能体是否能在具有连续状态和动作空间的动态交互式交通环境中，学习到安全且高效的变道策略？
• RQ2 嵌入车辆动力学知识对策略学习速度和稳定性有何影响？
• RQ3 所提出的闭式Q函数近似器在自动驾驶连续控制中是否优于演员-评论家或离散动作方法？
• RQ4 随着训练进行，智能体的驾驶行为在奖励和控制平滑性方面改善程度如何？
• RQ5 训练后的策略是否能在多个测试车辆上泛化，表明其鲁棒性和可迁移性？

主要发现

• 训练损失曲线显示明显收敛，表明预测Q值随时间与目标值高度一致。
• 单个车辆的总奖励稳步上升并在较高水平稳定，表明对惩罚的最小化能力得到提升。
• 每种模型在100辆测试车辆上的平均总奖励随训练步数持续增加，证实了策略性能的提升。
• 最终模型（400,000步）的偏航加速度和偏航速率曲线相比早期模型（40,000步）更平滑、变化更缓和，表明驾驶舒适性更优。
• 车辆智能体成功学习到一种平衡安全、舒适与效率的策略，符合奖励函数的定义。
• 该方法避免了额外的演员网络，降低了训练复杂度，同时实现了稳定且有效的策略学习。