[Paper Review] AttendLight: Universal Attention-Based Reinforcement Learning Model for Traffic Signal Control

Afshin Oroojlooy, Mohammadreza Nazari+2 more|arXiv (Cornell University)|Oct 12, 2020

Traffic control and management41 citations

TL;DR

AttendLight trains a universal RL model with two attention mechanisms to control traffic signals for intersections of varying topology, lanes, and phases, achieving strong performance across single- and multi-environment regimes.

ABSTRACT

We propose AttendLight, an end-to-end Reinforcement Learning (RL) algorithm for the problem of traffic signal control. Previous approaches for this problem have the shortcoming that they require training for each new intersection with a different structure or traffic flow distribution. AttendLight solves this issue by training a single, universal model for intersections with any number of roads, lanes, phases (possible signals), and traffic flow. To this end, we propose a deep RL model which incorporates two attention models. The first attention model is introduced to handle different numbers of roads-lanes; and the second attention model is intended for enabling decision-making with any number of phases in an intersection. As a result, our proposed model works for any intersection configuration, as long as a similar configuration is represented in the training set. Experiments were conducted with both synthetic and real-world standard benchmark data-sets. The results we show cover intersections with three or four approaching roads; one-directional/bi-directional roads with one, two, and three lanes; different number of phases; and different traffic flows. We consider two regimes: (i) single-environment training, single-deployment, and (ii) multi-environment training, multi-deployment. AttendLight outperforms both classical and other RL-based approaches on all cases in both regimes.

Motivation & Objective

• Motivate and develop a universal traffic signal control policy applicable to intersections with any topology and traffic pattern.
• Eliminate the need to redesign or retrain models for each new intersection by using attention mechanisms to handle variable inputs and outputs.
• Demonstrate that a single trained model can outperform traditional and RL-based baselines across synthetic and real-world datasets.

Proposed method

• Introduce AttendLight with two attention mechanisms: state-attention for phase representations and action-attention for next-phase decisions.
• Embed lane-level traffic features via a Conv1D embedding function and compute phase representations z_p^t as weighted sums over participating lanes.
• Use an LSTM to capture sequential phase information and derive the next-phase policy via action-attention over phase representations.
• Train with a variance-reduced REINFORCE algorithm under two regimes: single-env (train and deploy on the same intersection) and multi-env (train across multiple intersections and deploy broadly).
• Represent the state s^t as a set of lane characteristics s_l^t and use the negative intersection pressure as the reward to approximate travel-time minimization.

Experimental results

Research questions

• RQ1 Can a single, universal RL model achieve high-quality traffic signal control across intersections with different numbers of roads, lanes, and phases?
• RQ2 Does AttendLight generalize from training on multiple intersections to unseen intersection configurations without retraining?
• RQ3 How does AttendLight perform in single-environment versus multi-environment training regimes compared to traditional baselines?
• RQ4 What is the impact of the proposed attention mechanisms on handling variable input/output sizes in TSCP?
• RQ5 Is few-shot calibration effective to adapt the universal policy to a specific intersection?

Key findings

• AttendLight outperforms classical and several RL-based baselines on diverse intersections in both single-env and multi-env regimes.
• In the single-env regime, AttendLight achieves substantial improvements across 112 cases (e.g., 46% over FixedTime, 39% over MaxPressure, 34% over SOTL, 16% over DQTSC-M, and 9% over FRAP).
• In the multi-env regime, AttendLight yields about 13-15% ATT degradation on training vs testing sets, indicating good generalization and knowledge sharing between intersections.
• Across the 112 intersections, the multi-env policy generally outperforms FixedTime, MaxPressure, and SOTL, and is competitive with FRAP and often better than DQTSC-M.
• Few-shot calibration reduces the gap between multi-env and single-env from about 15% to 5% after 200 training episodes and to 3% after 1000 episodes, respectively.