[Paper Review] MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control

Liwen Zhu, Peixi Peng+3 more|arXiv (Cornell University)|Jan 4, 2021

Traffic control and management4 citations

TL;DR

MetaVIM proposes a meta-RL framework for decentralized traffic signal control that uses task-specific latent variables and intrinsic rewards to stabilize learning amid neighbor policy uncertainty. By modeling each intersection as a meta-learning task and enforcing observation/reward invariance via four decoders, MetaVIM achieves state-of-the-art performance on CityFlow with 724.21 mean travel time (vs. 1096.82 for baseline), demonstrating superior generalization and stability in diverse urban scenarios.

ABSTRACT

Traffic signal control aims to coordinate traffic signals across intersections to improve the traffic efficiency of a district or a city. Deep reinforcement learning (RL) has been applied to traffic signal control recently and demonstrated promising performance where each traffic signal is regarded as an agent. However, there are still several challenges that may limit its large-scale application in the real world. To make the policy learned from a training scenario generalizable to new unseen scenarios, a novel Meta Variationally Intrinsic Motivated (MetaVIM) RL method is proposed to learn the decentralized policy for each intersection that considers neighbor information in a latent way. Specifically, we formulate the policy learning as a meta-learning problem over a set of related tasks, where each task corresponds to traffic signal control at an intersection whose neighbors are regarded as the unobserved part of the state. Then, a learned latent variable is introduced to represent the task's specific information and is further brought into the policy for learning. In addition, to make the policy learning stable, a novel intrinsic reward is designed to encourage each agent's received rewards and observation transition to be predictable only conditioned on its own history. Extensive experiments conducted on CityFlow demonstrate that the proposed method substantially outperforms existing approaches and shows superior generalizability.

Motivation & Objective

• Address the challenge of learning effective decentralized policies in multi-intersection traffic control where neighbor agents' actions introduce uncertainty.
• Overcome the limitations of fixed or heuristic signal control by enabling adaptive, data-driven learning that generalizes across diverse traffic patterns.
• Improve policy stability and convergence in partially observable, non-stationary environments by modeling task-specific dynamics through learned latent variables.
• Enable rapid adaptation to new, unseen traffic scenarios through meta-learning over a set of related intersection control tasks.
• Design an intrinsic reward mechanism that encourages invariance in observations and rewards to neighbor policies, stabilizing training and improving generalization.

Proposed method

• Formulate traffic signal control as a meta-RL problem, where each intersection is a task with a unique latent variable encoding its historical trajectory and task-specific dynamics.
• Introduce a latent variable conditioned on past trajectories to represent task-specific reward, transition, and policy structures, enabling personalized policy adaptation.
• Deploy four decoders to predict next observations and rewards under four scenarios: current agent only, neighbor agents only, both, and neither, all conditioned on the latent variable.
• Design an intrinsic reward that minimizes the difference between predictions made with and without neighbor policies, promoting invariance and stabilizing learning.
• Use the intrinsic reward to balance exploration and exploitation by encouraging the agent to learn policies robust to neighbor behavior changes.
• Train the policy end-to-end using meta-RL objectives, enabling fast adaptation to new scenarios with minimal fine-tuning.

Figure 1: Illustration of the task uncertainty in multi-intersection control scenario. For intersection 1, the current policy is to choose North-South-Straight-Green conditioned on its local observation. However, under the same state, take the same action, when neighbour’s policy changes, transition

Experimental results

Research questions

• RQ1 Can meta-RL with task-specific latent variables improve generalization in decentralized traffic signal control across diverse, unseen urban environments?
• RQ2 How does modeling neighbor policy uncertainty through predictive decoders improve policy stability and convergence in multi-agent traffic control?
• RQ3 To what extent does an intrinsic reward based on observation and reward invariance enhance learning efficiency and performance in partially observable, non-stationary traffic environments?
• RQ4 Can a decentralized policy trained via meta-RL outperform centralized or independent RL baselines in large-scale, real-world traffic networks?
• RQ5 How does the proposed method handle distribution shifts in traffic patterns across different cities and traffic conditions?

Key findings

• MetaVIM achieves a mean travel time of 724.21 across four real-world cities (Hangzhou, Jinan, New York, Shenzhen), significantly outperforming the baseline model (1096.82) and existing methods like CoLight (767.75) and MetaLight (1147.33).
• In the real-world testing mode, MetaVIM reduces average travel time by 30.5% compared to the baseline, with gains of 44.8% in Hangzhou, 34.7% in Jinan, 28.6% in New York, and 30.4% in Shenzhen.
• The method demonstrates superior generalization, achieving strong performance across mixed low, mixed high, and real traffic patterns without retraining.
• The intrinsic reward mechanism reduces prediction error differences between neighbor-inclusive and neighbor-excluding scenarios, contributing to stable and robust policy learning.
• MetaVIM outperforms state-of-the-art methods such as CoLight, PressLight, and SOTL across all cities and traffic conditions, particularly in high-variability scenarios.
• Ablation studies confirm that both the latent variable and intrinsic reward components are essential: removing either leads to significant performance drops, validating their roles in generalization and stability.

Figure 2: The illustration of POMDPs in traffic signal control. We formulate the traffic signal control as a meta-learning problem across multiple POMDPs. In a POMDP, agent can only obtain its local observation and regard the neighbours’ policies as the unobserved part of the environment.