[Paper Review] Leveraging the Capabilities of Connected and Autonomous Vehicles and Multi-Agent Reinforcement Learning to Mitigate Highway Bottleneck Congestion

Paul Ha, Sikai Chen+4 more|arXiv (Cornell University)|Oct 11, 2020

Traffic control and management22 citations

TL;DR

This paper proposes a multi-agent reinforcement learning (MARL) framework using Graph Convolutional Networks (GCN) and Deep Deterministic Policy Gradient (DDPG) to enable connected and autonomous vehicles (CAVs) to collaboratively mitigate highway bottleneck congestion in mixed traffic. Even at just 10% CAV penetration, the RL-based controller significantly reduces congestion, outperforming rule-based approaches by effectively handling dynamic, topology-dependent traffic inputs.

ABSTRACT

Active Traffic Management strategies are often adopted in real-time to address such sudden flow breakdowns. When queuing is imminent, Speed Harmonization (SH), which adjusts speeds in upstream traffic to mitigate traffic showckwaves downstream, can be applied. However, because SH depends on driver awareness and compliance, it may not always be effective in mitigating congestion. The use of multiagent reinforcement learning for collaborative learning, is a promising solution to this challenge. By incorporating this technique in the control algorithms of connected and autonomous vehicle (CAV), it may be possible to train the CAVs to make joint decisions that can mitigate highway bottleneck congestion without human driver compliance to altered speed limits. In this regard, we present an RL-based multi-agent CAV control model to operate in mixed traffic (both CAVs and human-driven vehicles (HDVs)). The results suggest that even at CAV percent share of corridor traffic as low as 10%, CAVs can significantly mitigate bottlenecks in highway traffic. Another objective was to assess the efficacy of the RL-based controller vis-\\`a-vis that of the rule-based controller. In addressing this objective, we duly recognize that one of the main challenges of RL-based CAV controllers is the variety and complexity of inputs that exist in the real world, such as the information provided to the CAV by other connected entities and sensed information. These translate as dynamic length inputs which are difficult to process and learn from. For this reason, we propose the use of Graphical Convolution Networks (GCN), a specific RL technique, to preserve information network topology and corresponding dynamic length inputs. We then use this, combined with Deep Deterministic Policy Gradient (DDPG), to carry out multi-agent training for congestion mitigation using the CAV controllers.

Motivation & Objective

• To develop a scalable, real-time control strategy for mitigating highway bottleneck congestion using connected and autonomous vehicles (CAVs).
• To overcome the limitations of rule-based speed harmonization, which rely on human driver compliance and are thus inconsistent.
• To design a multi-agent reinforcement learning (MARL) controller that operates effectively in mixed traffic with human-driven vehicles (HDVs).
• To address the challenge of dynamic-length, topology-dependent inputs from connected entities using Graph Convolutional Networks (GCN).
• To evaluate the performance of the RL-based controller against rule-based controllers under realistic traffic conditions.

Proposed method

• The study employs a multi-agent reinforcement learning (MARL) framework where each CAV acts as an independent agent learning to optimize traffic flow.
• Graph Convolutional Networks (GCN) are used to process dynamic, variable-length inputs representing traffic connectivity and vehicle interactions, preserving spatial topology.
• The DDPG algorithm is applied for continuous action space control, enabling smooth speed adjustments by CAVs to mitigate shockwave propagation.
• The MARL framework is trained in a simulated mixed-traffic environment with both CAVs and HDVs to reflect real-world deployment conditions.
• Inputs include vehicle positions, speeds, and connectivity data from nearby vehicles, encoded via GCN to maintain structural information.
• The controller is trained to minimize congestion metrics such as total travel time and shockwave amplitude.

Experimental results

Research questions

• RQ1 Can a multi-agent reinforcement learning controller effectively reduce highway bottleneck congestion in mixed traffic with minimal CAV penetration?
• RQ2 How does the performance of the GCN-enhanced MARL controller compare to rule-based speed harmonization strategies?
• RQ3 Can the proposed framework handle dynamic, variable-length inputs from connected vehicle networks while preserving traffic topology information?
• RQ4 What is the minimum CAV penetration rate required for the RL controller to achieve significant congestion mitigation?
• RQ5 How robust is the RL-based controller to variations in traffic density and driver behavior in mixed-traffic scenarios?

Key findings

• At just 10% CAV penetration, the RL-based controller significantly reduces congestion, demonstrating high effectiveness even with low adoption rates.
• The GCN-based input processing enables the model to handle dynamic, topology-dependent traffic data more effectively than standard methods.
• The RL-based controller outperforms rule-based speed harmonization in reducing total travel time and mitigating shockwave amplitudes.
• The use of DDPG enables stable, continuous control actions that lead to smoother speed adjustments and reduced oscillations.
• The framework maintains performance across varying traffic densities and driver behavior patterns, indicating robustness.
• The integration of GCN with MARL allows for scalable and generalizable congestion mitigation in complex, real-world traffic networks.