[Paper Review] Multi-Vehicle Mixed-Reality Reinforcement Learning for Autonomous Multi-Lane Driving

Rupert Mitchell, Jenny Fletcher+2 more|arXiv (Cornell University)|Nov 26, 2019

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper presents a mixed reality reinforcement learning framework that enables safe, efficient training of autonomous driving policies in multi-vehicle, multi-lane environments by combining real-world policy adaptation with virtual vehicle interactions. By simulating collisions in a virtual domain while deploying the agent on a real vehicle, the method reduces collisions by over 80% after just a few mixed reality runs, demonstrating effective sim2real transfer for multi-agent autonomous driving.

ABSTRACT

Autonomous driving promises to transform road transport. Multi-vehicle and multi-lane scenarios, however, present unique challenges due to constrained navigation and unpredictable vehicle interactions. Learning-based methods---such as deep reinforcement learning---are emerging as a promising approach to automatically design intelligent driving policies that can cope with these challenges. Yet, the process of safely learning multi-vehicle driving behaviours is hard: while collisions---and their near-avoidance---are essential to the learning process, directly executing immature policies on autonomous vehicles raises considerable safety concerns. In this article, we present a safe and efficient framework that enables the learning of driving policies for autonomous vehicles operating in a shared workspace, where the absence of collisions cannot be guaranteed. Key to our learning procedure is a sim2real approach that uses real-world online policy adaptation in a mixed-reality setup, where other vehicles and static obstacles exist in the virtual domain. This allows us to perform safe learning by simulating (and learning from) collisions between the learning agent(s) and other objects in virtual reality. Our results demonstrate that, after only a few runs in mixed-reality, collisions are significantly reduced.

Motivation & Objective

• To address the safety challenges of training autonomous driving policies in multi-vehicle, multi-lane environments where collisions are common during learning.
• To enable safe policy learning by decoupling real-world deployment from collision-prone training via a mixed reality setup.
• To bridge the reality gap in reinforcement learning for autonomous vehicles by using real hardware for online policy adaptation while simulating interactions with other vehicles and obstacles.
• To demonstrate that end-to-end driving policies can be effectively learned and improved through minimal real-world interaction in a mixed reality environment.
• To provide a scalable, safe framework for training multiple autonomous agents in shared traffic spaces without risking physical damage.

Proposed method

• The framework uses a distributed reinforcement learning setup tailored for mixed reality, where one real DeepRacer agent learns in a physical track while 16 virtual vehicles and static obstacles are simulated.
• Virtual vehicles are modeled using standard traffic models (IDM and MOBIL) to generate realistic, dynamic interactions.
• The real vehicle's policy is updated online in real time using experience collected from interactions with virtual agents and obstacles in the simulation.
• Collisions are simulated in the virtual domain, allowing the agent to learn from near-misses and failures without physical risk.
• The system leverages a sim2real approach where the real vehicle's dynamics and sensor feedback are used to adapt the policy, minimizing the reality gap.
• The method employs model-free deep reinforcement learning with continuous action spaces, trained using off-policy algorithms to improve sample efficiency.

Experimental results

Research questions

• RQ1 Can a safe and efficient reinforcement learning framework be developed for multi-vehicle autonomous driving that avoids physical damage during policy training?
• RQ2 How effective is mixed reality reinforcement learning in reducing collisions and improving policy performance compared to simulation-only or real-world-only training?
• RQ3 To what extent can real-world online policy adaptation in a mixed reality setup reduce the reality gap in autonomous driving policy learning?
• RQ4 How do the dynamics of real hardware (e.g., DeepRacer) affect policy learning when interacting with faster, more responsive virtual agents?
• RQ5 Can a single real vehicle learn robust, collision-avoiding behaviors through interaction with virtual traffic in a shared multi-lane environment?

Key findings

• After only a few runs in the mixed reality environment, the number of collisions was significantly reduced, with the collision density distribution shifting sharply to the left.
• The post-training collision distribution was not only lower in mean but also narrower, indicating reduced variance and improved policy consistency.
• The learned policy achieved higher cumulative rewards in testing, confirming that the reinforcement learning process successfully improved performance.
• Qualitative results showed that the real vehicle transitioned from frequent collisions to near-collision avoidance after mixed reality training, demonstrating effective behavioral adaptation.
• The real vehicle's slower response time led to more cautious behavior, which reduced collision risk but also caused traffic congestion, suggesting a trade-off between safety and throughput.
• The use of high-entropy policies in simulation helped the agent generalize better to the real-world mixed reality setup, enabling faster adaptation without overfitting.