[论文解读] Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic
本文将混合交通条件下的高速公路匀道入口并道合流问题建模为一个去中心化多智能体强化学习问题，并引入一个具有动作屏蔽的可扩展 MARL 框架、一个基于优先级的安全监管器，以及课程学习，在安全性和效率方面优于基准方法。
On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.
研究动机与目标
- Formulate the mixed-traffic on-ramp merging problem as a decentralized MARL problem.
- Develop a scalable MARL framework with parameter sharing and local rewards for cooperation.
- Enhance learning efficiency and safety via action masking and a priority-based safety supervisor.
- Incorporate curriculum learning to efficiently master harder merging scenarios.
- Provide an open-source gym-like simulator and demonstrate superior performance over benchmarks.
提出的方法
- Model the on-ramp merging scenario as a partially observable MARL problem with AVs as agents.
- Use parameter sharing to enable scalability across variable agent counts and homogeneous policies.
- Incorporate local reward shaping to address credit assignment and foster cooperation among neighboring vehicles.
- Apply action masking to exclude invalid actions and improve learning stability.
- Introduce a priority-based safety supervisor that uses multi-step predictions (IDM/MOBIL HDV models) to prevent collisions.
- Utilize a kinematic bicycle model and low-level PID controllers to realize high-level MARL actions.
实验结果
研究问题
- RQ1Can a decentralized MARL framework with parameter sharing achieve safe and efficient highway on-ramp merging in mixed traffic?
- RQ2Does action masking and a priority-based safety supervisor improve learning efficiency and reduce collisions compared to baseline MARL methods?
- RQ3How does curriculum learning affect training efficiency and performance on harder merging scenarios?
- RQ4Does the proposed framework scale to dynamic topologies and varying numbers of AVs?
- RQ5What is the impact of local reward design on credit assignment and cooperative behavior?
主要发现
- The proposed MARL framework with action masking and a priority-based safety supervisor outperforms several state-of-the-art benchmarks in safety and efficiency across experiments.
- Local reward design mitigates credit assignment issues and improves cooperation among neighboring vehicles.
- Curriculum learning enables efficient mastery of harder merging tasks by building on simpler tasks.
- The safety supervisor reduces collision rates during training and accelerates learning, with practical real-time feasibility (approx. 28 ms per decision).
- The framework supports dynamic communication topologies and scales with the number of AVs, using parameter sharing to maintain a single policy across agents.
更好的研究，从现在开始
从阅读论文到最终审阅，大幅缩短您的研究时间。
无需绑定信用卡
本解读由 AI 生成，并经人工编辑审核。