[论文解读] Autonomous Ramp Merge Maneuver Based on Reinforcement Learning with Continuous Action Space

Pin Wang, Ching‐Yao Chan|arXiv (Cornell University)|Mar 25, 2018

Traffic control and management参考文献 10被引用 19

一句话总结

本文提出了一种基于强化学习（RL）的自主匝道并线系统，采用连续动作和状态空间，并使用二次Q函数近似以实现高效的策略学习。该方法在动态交通中实现了安全、平顺且及时的并线，证明了连续动作RL在高速公路以外复杂驾驶操作中的实用性。

ABSTRACT

Ramp merging is a critical maneuver for road safety and traffic efficiency. Most of the current automated driving systems developed by multiple automobile manufacturers and suppliers are typically limited to restricted access freeways only. Extending the automated mode to ramp merging zones presents substantial challenges. One is that the automated vehicle needs to incorporate a future objective (e.g. a successful and smooth merge) and optimize a long-term reward that is impacted by subsequent actions when executing the current action. Furthermore, the merging process involves interaction between the merging vehicle and its surrounding vehicles whose behavior may be cooperative or adversarial, leading to distinct merging countermeasures that are crucial to successfully complete the merge. In place of the conventional rule-based approaches, we propose to apply reinforcement learning algorithm on the automated vehicle agent to find an optimal driving policy by maximizing the long-term reward in an interactive driving environment. Most importantly, in contrast to most reinforcement learning applications in which the action space is resolved as discrete, our approach treats the action space as well as the state space as continuous without incurring additional computational costs. Our unique contribution is the design of the Q-function approximation whose format is structured as a quadratic function, by which simple but effective neural networks are used to estimate its coefficients. The results obtained through the implementation of our training platform demonstrate that the vehicle agent is able to learn a safe, smooth and timely merging policy, indicating the effectiveness and practicality of our approach.

研究动机与目标

• 开发一种可将自动驾驶能力延伸至高速公路以外区域的自主匝道并线系统。
• 解决在存在相互作用车辆的动态交通中长期奖励优化的挑战。
• 通过建模连续控制动作，克服离散动作RL的局限，实现更平滑的并线。
• 设计一种高效的Q函数近似方法，避免在连续动作空间中产生高计算成本。

提出的方法

• 作者采用具有连续动作和状态空间的深度强化学习，将并线车辆建模为智能体。
• 使用二次函数结构化Q函数近似，实现在不增加计算开销的前提下高效学习。
• 利用神经网络估计二次Q函数的系数，实现端到端的策略学习。
• 环境模拟了真实的交通交互，包括周围车辆的协作与对抗行为。
• 智能体被训练以最大化反映并线安全性、平顺性与及时性的长期奖励。
• 该方法避免了动作的离散化，保留了车辆控制输入（如转向与加速度）的自然连续性。

实验结果

研究问题

• RQ1 连续动作强化学习能否在动态交通中有效学习到安全且平顺的匝道并线策略？
• RQ2 二次Q函数近似如何提升连续控制强化学习中的样本效率并降低计算成本？
• RQ3 与离散化相比，将动作和状态建模为连续形式在自动驾驶并线中能带来哪些性能提升？
• RQ4 在并线过程中，智能体如何应对周围车辆的对抗性或不可预测行为？
• RQ5 所学策略在不同交通密度的真实并线场景中具有多大程度的泛化能力？

主要发现

• 车辆智能体通过在模拟交互环境中的训练，成功学习到安全且平顺的并线策略。
• 连续动作方法相比离散动作替代方案，实现了更自然且精确的控制。
• 二次Q函数近似在保持高学习效率的同时降低了计算成本。
• 系统在各种交通条件下均实现了及时并线，且未牺牲安全性或舒适性。
• 结果表明，连续动作RL在复杂自动驾驶任务中具有实际可行性和有效性。