[Paper Review] Multi-agent Reinforcement Learning for Networked System Control

Tianshu Chu, Sandeep Chinchali+1 more|arXiv (Cornell University)|Apr 3, 2020

Traffic control and management23 references62 citations

TL;DR

The paper formulates networked multi-agent reinforcement learning as a spatiotemporal MDP, introduces a spatial discount factor, and proposes NeurComm to enable adaptive neural communication for improved learning and control in traffic signal control and cooperative adaptive cruise control tasks.

ABSTRACT

This paper considers multi-agent reinforcement learning (MARL) in networked system control. Specifically, each agent learns a decentralized control policy based on local observations and messages from connected neighbors. We formulate such a networked MARL (NMARL) problem as a spatiotemporal Markov decision process and introduce a spatial discount factor to stabilize the training of each local agent. Further, we propose a new differentiable communication protocol, called NeurComm, to reduce information loss and non-stationarity in NMARL. Based on experiments in realistic NMARL scenarios of adaptive traffic signal control and cooperative adaptive cruise control, an appropriate spatial discount factor effectively enhances the learning curves of non-communicative MARL algorithms, while NeurComm outperforms existing communication protocols in both learning efficiency and control performance.

Motivation & Objective

• Formulate NMARL for networked control with neighborhood communication under offline training.
• Introduce a spatial discount factor to stabilize learning in decentralized settings.
• Propose NeurComm, a differentiable communication protocol, to reduce information loss and non-stationarity.
• Evaluate NMARL and NeurComm in adaptive traffic signal control and cooperative adaptive cruise control scenarios.

Proposed method

• Model the networked system as a decentralized spatiotemporal MDP with neighborhood communication.
• Define a spatial discount factor alpha to weight rewards by distance between agents.
• Develop NeurComm, a differentiable message encoding/decoding scheme that includes state and policy fingerprints in messages.
• Derive actor-critic (A2C) updates for each agent using local observations and neighbor messages (equations for policy loss and value loss).
• Compare communicative and non-communicative MARL baselines on ATSC and CACC tasks using offline rollout training and offline-global information exposure.

Experimental results

Research questions

• RQ1 How does a spatial discount factor affect learning stability and performance for non-communicative MARL in networked control?
• RQ2 Can a learnable neural communication protocol (NeurComm) improve learning efficiency and control performance over existing protocols in NMARL?
• RQ3 How do NMARL approaches perform on realistic NSC tasks such as adaptive traffic signal control and cooperative adaptive cruise control under neighborhood communication?
• RQ4 What are the trade-offs between communicative and non-communicative MARL in terms of sample efficiency, robustness, and execution performance?

Key findings

• A spatial discount factor generally improves learning curves for non-communicative MARL, with alpha values often near 0.9 or higher.
• NeurComm outperforms existing communication protocols in learning efficiency and control performance on challenging NSC scenarios.
• NeurComm remains competitive or superior in ATSC tasks and often outperforms alternatives in more complex Monaco network settings.
• Communicative policies can yield faster performance on some metrics but may struggle with overfitting in heterogenous networks, depending on scenario.
• Best spatial discount factors and communication strategies vary by scenario, with NeurComm showing strong performance in ATSC tasks.
• Across settings, a tuned spatial discount factor alpha correlates with task difficulty and level of coordination required.