[Paper Review] Virtual to Real Reinforcement Learning for Autonomous Driving

Xinlei Pan, Yurong You+2 more|arXiv (Cornell University)|Apr 13, 2017

Reinforcement Learning in Robotics29 references149 citations

TL;DR

The paper proposes a realistic translation network that converts virtual simulator frames into realistic images via scene parsing, enabling reinforcement learning agents trained in simulation to transfer effectively to real-world driving.

ABSTRACT

Reinforcement learning is considered as a promising direction for driving policy learning. However, training autonomous driving vehicle with reinforcement learning in real environment involves non-affordable trial-and-error. It is more desirable to first train in a virtual environment and then transfer to the real environment. In this paper, we propose a novel realistic translation network to make model trained in virtual environment be workable in real world. The proposed network can convert non-realistic virtual image input into a realistic one with similar scene structure. Given realistic frames as input, driving policy trained by reinforcement learning can nicely adapt to real world driving. Experiments show that our proposed virtual to real (VR) reinforcement learning (RL) works pretty well. To our knowledge, this is the first successful case of driving policy trained by reinforcement learning that can adapt to real world driving data.

Motivation & Objective

• Motivate safe, cost-effective policy learning by training in virtual environments before real-world deployment.
• Develop a two-stage image translation pipeline that preserves scene parsing structure while converting virtual images to realistic ones.
• Demonstrate that RL agents trained with translated realistic images outperform those trained purely in virtual domains or with domain randomization.
• Compare against supervised learning baselines to assess data-efficiency and generalization in real-world driving tasks.

Proposed method

• Introduce a two-module realistic translation network: virtual-to-parsing (virtual images to scene parsing maps) and parsing-to-real (parsing maps to realistic images).
• Train the translation networks with conditional GAN objectives augmented by an L1 loss to reduce blurring (G* = arg min_G max_D L_cGAN + lambda L1).
• Use SegNet-style semantic segmentation to obtain scene parsing representations that serve as an intermediate domain bridging virtual and real images.
• Train an RL agent with Asynchronous Advantage Actor-Critic (A3C) using realistic translated frames as state inputs to learn driving policies.
• Evaluate steering/action accuracy by mapping real-world steering labels to simulator actions for fair comparisons.
• Benchmark against a purely virtual (B-RL) agent and a supervised (SV) model using real driving data.

Experimental results

Research questions

• RQ1 Can translating virtual images to realistic frames via a scene-parsing intermediate representation enable RL policies trained in simulation to transfer to real-world driving?
• RQ2 Does VR RL with parsed-to-real image synthesis outperform domain-randomized RL and purely supervised baselines on real driving data?
• RQ3 What is the impact of using a segmentation-based bridge versus direct virtual-to-real mappings for policy transfer?
• RQ4 How does the proposed method perform in transfer learning across different virtual driving environments?

Key findings

• The VR RL approach yields higher action prediction accuracy than the baseline RL that uses only virtual inputs (43.40% vs 28.33%).
• Supervised learning achieves the best accuracy (53.60%), but requires large amounts of labeled data.
• VR RL outperforms domain randomization RL in transfer experiments between virtual environments.
• Image-to-segmentation-to-image translation preserves scene structure enabling realistic frames suitable for RL training.
• Segmentation-based intermediates make it feasible to translate between virtual and real domains without paired virtual-real data.
• The method demonstrates a first successful case of driving policy trained by RL that adapts to real-world driving data.