[论文解读] Graph Reinforcement Learning Application to Co-operative Decision-Making in Mixed Autonomy Traffic: Framework, Survey, and Challenges

Qi Liu, Xueyuan Li+6 more|arXiv (Cornell University)|Nov 6, 2022

Traffic control and management被引用 7

一句话总结

本文提出了一种图强化学习（GRL）框架，用于混合自动化交通中的多智能体决策，将车辆建模为节点，交互作用建模为边，以捕捉动态的车辆关系。GRL在复杂、高维的交通环境中，通过实现更有效的合作行为，优于深度强化学习（DRL）方法。

ABSTRACT

Proper functioning of connected and automated vehicles (CAVs) is crucial for the safety and efficiency of future intelligent transport systems. Meanwhile, transitioning to fully autonomous driving requires a long period of mixed autonomy traffic, including both CAVs and human-driven vehicles. Thus, collaboration decision-making for CAVs is essential to generate appropriate driving behaviors to enhance the safety and efficiency of mixed autonomy traffic. In recent years, deep reinforcement learning (DRL) has been widely used in solving decision-making problems. However, the existing DRL-based methods have been mainly focused on solving the decision-making of a single CAV. Using the existing DRL-based methods in mixed autonomy traffic cannot accurately represent the mutual effects of vehicles and model dynamic traffic environments. To address these shortcomings, this article proposes a graph reinforcement learning (GRL) approach for multi-agent decision-making of CAVs in mixed autonomy traffic. First, a generic and modular GRL framework is designed. Then, a systematic review of DRL and GRL methods is presented, focusing on the problems addressed in recent research. Moreover, a comparative study on different GRL methods is further proposed based on the designed framework to verify the effectiveness of GRL methods. Results show that the GRL methods can well optimize the performance of multi-agent decision-making for CAVs in mixed autonomy traffic compared to the DRL methods. Finally, challenges and future research directions are summarized. This study can provide a valuable research reference for solving the multi-agent decision-making problems of CAVs in mixed autonomy traffic and can promote the implementation of GRL-based methods into intelligent transportation systems. The source code of our work can be found at https://github.com/Jacklinkk/Graph_CAVs.

研究动机与目标

• 解决现有深度强化学习（DRL）方法在建模混合自动化交通中多智能体交互方面的局限性。
• 开发一种通用、模块化的图强化学习（GRL）框架，用于表示车辆交互并优化合作决策。
• 系统性回顾近期用于混合交通环境中CAV决策的DRL与GRL方法。
• 通过对比研究验证GRL在多智能体CAV场景中相对于DRL的有效性。
• 识别基于GRL的智能交通系统中多智能体决策的关键挑战与未来研究方向。

提出的方法

• 将混合自动化交通建模为图结构，其中每个联网与自动化车辆（CAV）作为节点，车辆间交互作为边。
• 采用图神经网络（GNNs）从交通图中提取空间与关系特征，用于策略学习。
• 将GNN处理后的特征集成到深度Q网络（DQN）或类似策略网络中，以生成合作驾驶行为。
• 设计模块化框架，支持多种GNN架构与强化学习算法的灵活集成。
• 采用多智能体强化学习设置，实现集中式训练与分布式执行，以优化联合策略。
• 在GitHub上实现并开源该框架，以确保可复现性与社区使用。

实验结果

研究问题

• RQ1 图强化学习如何有效建模混合自动化交通中动态的多智能体交互？
• RQ2 在合作CAV决策中，GRL与基于DRL的方法在性能上存在哪些关键差异？
• RQ3 GRL方法如何处理混合交通环境中典型的高维状态与动作空间？
• RQ4 在现实混合自动化交通中应用GRL时，面临的关键挑战有哪些，包括不确定性、通信范围与车辆建模等问题？
• RQ5 如何扩展GRL以支持全局协调，同时保持本地决策的高效性？

主要发现

• GRL方法在优化混合自动化交通中CAV的多智能体决策方面，显著优于传统的基于DRL的方法。
• 基于图的表示能有效捕捉车辆间的依赖关系与动态交通交互，从而实现更合作、更安全的驾驶行为。
• 所提出的GRL框架在汇入与交叉口控制等场景中表现出更优性能，有效减少延误与燃油消耗。
• 对比研究证实，在复杂、交互性强的交通环境中，基于GRL的模型相比DRL模型具有更好的协调性与稳定性。
• 该框架具备可扩展性，支持与多种GNN和RL组件的集成，可适应未来新场景的演进。
• 在GitHub上的开源实现促进了可复现性，并推动了GRL在智能交通系统领域的进一步研究。