[Paper Review] Reinforcement Learning Based Oscillation Dampening: Scaling up Single-Agent RL algorithms to a 100 AV highway field operational test

Kathy Jang, Nathan Lichtlé+11 more|arXiv (Cornell University)|Feb 26, 2024

Traffic control and management5 citations

TL;DR

The paper documents the deployment of RL-based controllers on the MegaVanderTest, the largest AV field test to smooth traffic, and analyzes simulation-to-deployment results, safety considerations, and robustness.

ABSTRACT

In this article, we explore the technical details of the reinforcement learning (RL) algorithms that were deployed in the largest field test of automated vehicles designed to smooth traffic flow in history as of 2023, uncovering the challenges and breakthroughs that come with developing RL controllers for automated vehicles. We delve into the fundamental concepts behind RL algorithms and their application in the context of self-driving cars, discussing the developmental process from simulation to deployment in detail, from designing simulators to reward function shaping. We present the results in both simulation and deployment, discussing the flow-smoothing benefits of the RL controller. From understanding the basics of Markov decision processes to exploring advanced techniques such as deep RL, our article offers a comprehensive overview and deep dive of the theoretical foundations and practical implementations driving this rapidly evolving field. We also showcase real-world case studies and alternative research projects that highlight the impact of RL controllers in revolutionizing autonomous driving. From tackling complex urban environments to dealing with unpredictable traffic scenarios, these intelligent controllers are pushing the boundaries of what automated vehicles can achieve. Furthermore, we examine the safety considerations and hardware-focused technical details surrounding deployment of RL controllers into automated vehicles. As these algorithms learn and evolve through interactions with the environment, ensuring their behavior aligns with safety standards becomes crucial. We explore the methodologies and frameworks being developed to address these challenges, emphasizing the importance of building reliable control systems for automated vehicles.

Motivation & Objective

• Demonstrate the deployment of single-agent RL controllers for traffic smoothing in a large-scale AV field test (100 AVs) on a real highway.
• Explain the transition from simulation to hardware deployment and the role of data-driven, fast simulators built from real trajectory data.
• Assess the flow-smoothing benefits and safety considerations of RL controllers in mixed-autonomy traffic.
• Compare RL controllers to traditional baselines (e.g., FollowerStopper) in both simulation and field deployment.

Proposed method

• Formulate the control problem within the RL framework for partially observable MDPs.
• Use policy gradient methods with Proximal Policy Optimization (PPO) to train controllers.
• Model human drivers with the Intelligent Driver Model (IDM) and introduce string-unstable dynamics to generate realistic traffic waves.
• Develop a data-driven one-lane simulator built from I-24 trajectory data to enable fast training and evaluation.
• Deploy RL controllers on AVs with a central server–vehicle communication scheme in the MegaVanderTest field trial.

Experimental results

Research questions

• RQ1 How do single-agent RL controllers perform in large-scale, real-world highway deployments with mixed-autonomy traffic?
• RQ2 Can RL-trained controllers trained in simulation transfer effectively to hardware deployment and real traffic, and how do they compare to hand-designed baselines?
• RQ3 What are the safety, robustness, and generalization properties of RL controllers under varying downstream speed limits and penetration rates?
• RQ4 What are the energy and throughput impacts of RL-based wave-damping in high-density traffic?
• RQ5 What is the role of data-driven simulators in enabling rapid, realistic training and evaluation for field-ready RL controllers?

Key findings

• The study documents the largest field test of automated vehicles aimed at smoothing traffic flow (as of 2023).
• A data-driven, fast simulator built from I-24 trajectory data enables rapid training and evaluation of RL controllers with minimal sim-to-real gap.
• In a representative wave-damping scenario, an RL controller trained at 10% penetration and downstream speed limit of 5 m/s achieved a 25% reduction in average fuel consumption compared to the uncontrolled baseline.
• The RL controller demonstrated robustness to changes in penetration rate and downstream speed limit beyond its training domain.
• Compared to the FollowerStopper baseline, the RL controller can generalize and outperform in various training and evaluation settings without requiring knowledge of the downstream speed limit.
• The deployment framework involves centralized planning and vehicle-level controllers, with safety measures and PMDP considerations discussed for real-world operation.