[Paper Review] Dampen the Stop-and-Go Traffic with Connected and Automated Vehicles -- A Deep Reinforcement Learning Approach

Liming Jiang, Yuanchang Xie+3 more|arXiv (Cornell University)|May 17, 2020

Traffic control and management25 references4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL) controller for connected automated vehicles (CAVs) to suppress stop-and-go traffic waves in a platoon. By positioning a single CAV at the second position and training it to stabilize speed oscillations, the method reduces speed fluctuations by 54% in the CAV and 8–28% in following human-driven vehicles, while also lowering fuel consumption significantly.

ABSTRACT

Stop-and-go traffic poses many challenges to tranportation system, but its formation and mechanism are still under exploration.however, it has been proved that by introducing Connected Automated Vehicles(CAVs) with carefully designed controllers one could dampen the stop-and-go waves in the vehicle fleet. Instead of using analytical model, this study adopts reinforcement learning to control the behavior of CAV and put a single CAV at the 2nd position of a vehicle fleet with the purpose to dampen the speed oscillation from the fleet leader and help following human drivers adopt more smooth driving behavior. The result show that our controller could decrease the spped oscillation of the CAV by 54% and 8%-28% for those following human-driven vehicles. Significant fuel consumption savings are also observed. Additionally, the result suggest that CAVs may act as a traffic stabilizer if they choose to behave slightly altruistically.

Motivation & Objective

• Address the persistent challenge of stop-and-go traffic waves in urban and highway traffic systems.
• Investigate whether reinforcement learning can outperform analytical control models in stabilizing traffic flow.
• Explore the role of a single CAV as a traffic stabilizer in mixed human-CAV platoons.
• Evaluate the impact of altruistic behavior in CAVs on overall traffic smoothness and fuel efficiency.
• Demonstrate the feasibility of using DRL to design scalable, adaptive traffic control strategies for connected and automated vehicles.

Proposed method

• Employ deep Q-network (DQN) reinforcement learning to train a CAV to regulate its speed based on local traffic states.
• Position the CAV at the second position in a vehicle platoon, behind the fleet leader, to influence downstream vehicles.
• Define a reward function that penalizes speed deviations and accelerations, promoting smooth driving behavior.
• Use a continuous action space with a dueling DQN architecture to improve policy learning and value estimation.
• Train the agent in a simulated environment with realistic traffic dynamics and human driver behavior models.
• Enable the CAV to learn altruistic behavior by adjusting the reward function to prioritize platoon-wide stability over individual performance.

Experimental results

Research questions

• RQ1 Can a DRL-based controller effectively suppress stop-and-go traffic waves in a mixed human-CAV platoon?
• RQ2 How does the placement of a single CAV at the second position affect the stability of the entire vehicle fleet?
• RQ3 To what extent can a CAV reduce speed oscillations in both itself and following human-driven vehicles?
• RQ4 What is the impact of altruistic behavior in the CAV controller on fuel consumption and traffic smoothness?
• RQ5 How does the DRL approach compare to traditional analytical control methods in damping traffic waves?

Key findings

• The DRL controller reduced speed oscillation in the CAV by 54% compared to baseline scenarios.
• Following human-driven vehicles experienced a 8% to 28% reduction in speed oscillation due to the presence of the CAV.
• Significant fuel consumption savings were observed, indicating improved energy efficiency in the platoon.
• The results suggest that CAVs can act as effective traffic stabilizers when trained to behave slightly altruistically.
• The DRL approach outperformed traditional analytical models in adapting to complex, dynamic traffic conditions.
• The controller demonstrated robustness across varying traffic densities and driver behavior patterns in simulation.