[Paper Review] Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

Dong Chen, Hajidavalloo, Mohammad+5 more|arXiv (Cornell University)|May 12, 2021

Traffic control and managementEngineering45 references36 citations

TL;DR

The paper formulates highway on-ramp merging in mixed traffic as a decentralized multi-agent reinforcement learning problem and introduces a scalable MARL framework with action masking, a priority-based safety supervisor, and curriculum learning, achieving superior safety and efficiency over benchmarks.

ABSTRACT

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.

Motivation & Objective

• Formulate the mixed-traffic on-ramp merging problem as a decentralized MARL problem.
• Develop a scalable MARL framework with parameter sharing and local rewards for cooperation.
• Enhance learning efficiency and safety via action masking and a priority-based safety supervisor.
• Incorporate curriculum learning to efficiently master harder merging scenarios.
• Provide an open-source gym-like simulator and demonstrate superior performance over benchmarks.

Proposed method

• Model the on-ramp merging scenario as a partially observable MARL problem with AVs as agents.
• Use parameter sharing to enable scalability across variable agent counts and homogeneous policies.
• Incorporate local reward shaping to address credit assignment and foster cooperation among neighboring vehicles.
• Apply action masking to exclude invalid actions and improve learning stability.
• Introduce a priority-based safety supervisor that uses multi-step predictions (IDM/MOBIL HDV models) to prevent collisions.
• Utilize a kinematic bicycle model and low-level PID controllers to realize high-level MARL actions.

Figure 1: Illustration of the considered on-ramp merging traffic scenario. CAVs (blue) and HDVs (green) coexist on both ramp and through lanes.

Experimental results

Research questions

• RQ1 Can a decentralized MARL framework with parameter sharing achieve safe and efficient highway on-ramp merging in mixed traffic?
• RQ2 Does action masking and a priority-based safety supervisor improve learning efficiency and reduce collisions compared to baseline MARL methods?
• RQ3 How does curriculum learning affect training efficiency and performance on harder merging scenarios?
• RQ4 Does the proposed framework scale to dynamic topologies and varying numbers of AVs?
• RQ5 What is the impact of local reward design on credit assignment and cooperative behavior?

Key findings

• The proposed MARL framework with action masking and a priority-based safety supervisor outperforms several state-of-the-art benchmarks in safety and efficiency across experiments.
• Local reward design mitigates credit assignment issues and improves cooperation among neighboring vehicles.
• Curriculum learning enables efficient mastery of harder merging tasks by building on simpler tasks.
• The safety supervisor reduces collision rates during training and accelerates learning, with practical real-time feasibility (approx. 28 ms per decision).
• The framework supports dynamic communication topologies and scales with the number of AVs, using parameter sharing to maintain a single policy across agents.

Figure 2: Schematics of system and simulation setup.