[Paper Review] Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control

Tianshu Chu, Jie Wang+2 more|arXiv (Cornell University)|Mar 11, 2019

Traffic control and management18 references4 citations

TL;DR

This paper proposes a decentralized multi-agent advantage actor-critic (MA2C) framework for large-scale traffic signal control, enhancing observability via neighbor fingerprints and reducing learning difficulty with a spatial discount factor. Evaluated on synthetic and real-world Monaco traffic networks, MA2C outperforms independent A2C and Q-learning baselines in optimality, robustness, and sample efficiency, achieving lower and more sustainable delays and improved network capacity utilization.

ABSTRACT

Reinforcement learning (RL) is a promising data-driven approach for adaptive traffic signal control (ATSC) in complex urban traffic networks, and deep neural networks further enhance its learning power. However, centralized RL is infeasible for large-scale ATSC due to the extremely high dimension of the joint action space. Multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Most existing studies in MARL focus on designing efficient communication and coordination among traditional Q-learning agents. This paper presents, for the first time, a fully scalable and decentralized MARL algorithm for the state-of-the-art deep RL agent: advantage actor critic (A2C), within the context of ATSC. In particular, two methods are proposed to stabilize the learning procedure, by improving the observability and reducing the learning difficulty of each local agent. The proposed multi-agent A2C is compared against independent A2C and independent Q-learning algorithms, in both a large synthetic traffic grid and a large real-world traffic network of Monaco city, under simulated peak-hour traffic dynamics. Results demonstrate its optimality, robustness, and sample efficiency over other state-of-the-art decentralized MARL algorithms.

Motivation & Objective

• To address the scalability and partial observability challenges in large-scale adaptive traffic signal control (ATSC) using multi-agent reinforcement learning (MARL).
• To develop a fully decentralized, scalable, and stable MARL algorithm based on the state-of-the-art advantage actor-critic (A2C) framework for ATSC.
• To improve local agent observability and reduce learning difficulty in decentralized MARL settings, where agents have limited communication and partial views of the environment.
• To evaluate the proposed method against independent A2C and Q-learning baselines in both synthetic and real-world traffic networks under realistic peak-hour dynamics.
• To demonstrate superior performance in optimality, robustness, and sample efficiency compared to existing decentralized MARL approaches.

Proposed method

• Proposes a multi-agent A2C (MA2C) algorithm that extends independent A2C to cooperative MARL, enabling decentralized control of individual intersections.
• Introduces neighbor fingerprints—shared representations of neighboring intersection states—to improve local observability and coordination.
• Incorporates a spatial discount factor in the reward function to reduce the temporal credit assignment problem and lower learning difficulty for local agents.
• Uses deep neural networks to parameterize both policy and value functions in the A2C framework, enabling function approximation in high-dimensional state-action spaces.
• Employs experience replay and bootstrapped sampling for stable training, while maintaining on-policy learning to reduce variance.
• Applies normalization to state and reward signals to stabilize training, particularly in high-density traffic scenarios.

Experimental results

Research questions

• RQ1 Can a decentralized multi-agent A2C framework achieve stable and scalable learning in large-scale traffic signal control under partial observability?
• RQ2 How do neighbor fingerprints and spatial discounting improve observability and learning efficiency in multi-agent traffic control?
• RQ3 Does the proposed MA2C algorithm outperform independent A2C and Q-learning baselines in terms of delay reduction, queue management, and network capacity utilization?
• RQ4 How robust is the MA2C framework under realistic, time-varying, and stochastic traffic dynamics in real-world urban networks?
• RQ5 To what extent does the algorithm maintain performance when faced with noisy or delayed sensor measurements in real deployments?

Key findings

• MA2C achieved faster and more stable training convergence compared to IA2C and IQL-DNN in both synthetic and Monaco traffic networks.
• In the Monaco network, MA2C reduced average intersection delay to lower and more sustainable levels than IA2C and greedy policies, particularly after peak congestion.
• MA2C maintained network flows closer to the 'sweet spot' of the macroscopic fundamental diagram, maximizing network capacity utilization under saturation.
• The algorithm outperformed all baselines in minimizing maximum queue length and trip delay, demonstrating superior optimality and robustness.
• Independent Q-learning and IQL-DNN failed to converge or perform reliably, while IA2C showed instability despite reasonable performance in mid-training.
• The use of neighbor fingerprints and spatial discounting significantly enhanced learning stability and performance, especially in partially observable environments.