[Paper Review] Attacking Deep Reinforcement Learning-Based Traffic Signal Control Systems with Colluding Vehicles

Ao Qu, Yihong Tang+1 more|arXiv (Cornell University)|Nov 4, 2021

Adversarial Robustness in Machine Learning31 references4 citations

TL;DR

This paper proposes CollusionVeh, a deep reinforcement learning-based framework enabling a group of vehicles to cooperatively send falsified traffic data to manipulate DRL-powered traffic signal control systems, significantly reducing their collective travel time. The method uses shared embeddings and communication mechanisms to coordinate vehicles, achieving up to 65.7% improvement in cumulative rewards (correlated with time saved) with minimal action space.

ABSTRACT

The rapid advancements of Internet of Things (IoT) and artificial intelligence (AI) have catalyzed the development of adaptive traffic signal control systems (ATCS) for smart cities. In particular, deep reinforcement learning (DRL) methods produce the state-of-the-art performance and have great potentials for practical applications. In the existing DRL-based ATCS, the controlled signals collect traffic state information from nearby vehicles, and then optimal actions (e.g., switching phases) can be determined based on the collected information. The DRL models fully "trust" that vehicles are sending the true information to the signals, making the ATCS vulnerable to adversarial attacks with falsified information. In view of this, this paper first time formulates a novel task in which a group of vehicles can cooperatively send falsified information to "cheat" DRL-based ATCS in order to save their total travel time. To solve the proposed task, we develop CollusionVeh, a generic and effective vehicle-colluding framework composed of a road situation encoder, a vehicle interpreter, and a communication mechanism. We employ our method to attack established DRL-based ATCS and demonstrate that the total travel time for the colluding vehicles can be significantly reduced with a reasonable number of learning episodes, and the colluding effect will decrease if the number of colluding vehicles increases. Additionally, insights and suggestions for the real-world deployment of DRL-based ATCS are provided. The research outcomes could help improve the reliability and robustness of the ATCS and better protect the smart mobility systems.

Motivation & Objective

• To investigate the vulnerability of DRL-based adaptive traffic signal control systems (ATCS) to coordinated adversarial attacks using falsified vehicle-reported data.
• To model a realistic attack scenario where a group of connected vehicles collude to minimize their total travel time by manipulating signal decisions.
• To develop a generic, interpretable framework that enables colluding vehicles to coordinate their actions using shared representations and communication.
• To evaluate the effectiveness of coordinated attacks versus individual greedy strategies under varying collusion sizes and action space constraints.

Proposed method

• Designs a multi-agent DRL framework where colluding vehicles share policy parameters to generate scenario embeddings, enabling joint understanding of traffic conditions.
• Employs a road situation encoder to extract global traffic state representations from vehicle observations, enabling shared context awareness.
• Uses a vehicle interpreter to map individual vehicle states into actionable strategies based on shared embeddings and communication.
• Introduces a communication mechanism allowing vehicles to exchange information to coordinate actions and avoid conflicts during signal phase requests.
• Applies multi-agent actor-critic (MA2C) training to optimize the collective reward, defined as inverse of total travel time.
• Uses parameter sharing across vehicles to improve sample efficiency and enable generalization across different traffic scenarios.

Experimental results

Research questions

• RQ1 Can a group of vehicles cooperatively manipulate DRL-based traffic signal control systems by sending falsified traffic data to reduce their collective travel time?
• RQ2 How does the performance of coordinated collusion compare to individual vehicles acting greedily in sending falsified information?
• RQ3 How does the size of the collusion group affect the average and total travel time savings?
• RQ4 How does the action space (number of possible signal phase requests) influence the effectiveness of the attack?
• RQ5 What are the key system-level vulnerabilities in DRL-based ATCS that enable such coordinated manipulation?

Key findings

• Colluding vehicles using CollusionVeh achieved a 65.7% improvement in cumulative reward (indicating time savings) even with a minimal action space of {0,1,2,3}, demonstrating high effectiveness with low complexity.
• The total travel time for the collusion group decreased significantly with more learning episodes, indicating that the attack becomes effective after sufficient policy inference.
• As the number of colluding vehicles increased, the average individual time saved decreased, indicating diminishing returns due to intra-group competition for signal phases.
• The coordinated strategy in CollusionVeh outperformed a greedy baseline that always requested the maximum possible action, especially as action space expanded.
• The attack was effective even when the DRL policy was hidden and unknown to the vehicles, showing that policy inference via imitation learning is feasible in this setting.
• The study reveals that DRL-based ATCS are vulnerable to coordinated data falsification, especially when policies are not updated frequently.