[Paper Review] Transferring Autonomous Driving Knowledge on Simulated and Real Intersections

David Isele, Akansel Cosgun|arXiv (Cornell University)|Nov 30, 2017

Reinforcement Learning in Robotics23 references14 citations

TL;DR

This paper investigates knowledge transfer in autonomous intersection navigation using deep reinforcement learning, demonstrating that fine-tuning a pre-trained DQN on a new intersection type significantly improves performance over random initialization. It further shows that sim-to-real transfer is effective, with pre-training in simulation accelerating real-world learning and enhancing generalization.

ABSTRACT

We view intersection handling on autonomous vehicles as a reinforcement learning problem, and study its behavior in a transfer learning setting. We show that a network trained on one type of intersection generally is not able to generalize to other intersections. However, a network that is pre-trained on one intersection and fine-tuned on another performs better on the new task compared to training in isolation. This network also retains knowledge of the prior task, even though some forgetting occurs. Finally, we show that the benefits of fine-tuning hold when transferring simulated intersection handling knowledge to a real autonomous vehicle.

Motivation & Objective

• To evaluate whether knowledge learned in one intersection type generalizes to others in autonomous driving.
• To investigate if fine-tuning a pre-trained policy on a new intersection improves learning efficiency and performance compared to random initialization.
• To assess reverse transfer: whether knowledge of the original task is retained after fine-tuning on a new task.
• To examine the feasibility of transferring policies trained in simulation to real-world autonomous vehicles.
• To determine if transfer learning can reduce sample complexity and improve robustness in real-world deployment.

Proposed method

• Formulates intersection handling as a Markov Decision Process (MDP) with state, action, reward, and transition dynamics.
• Uses Deep Q-Networks (DQNs) to represent the Q-value function for state-action pairs in intersection navigation.
• Employs direct copy, fine-tuning, and reverse transfer protocols to evaluate knowledge transfer between different intersection types.
• Applies transfer from simulated environments to real-world autonomous vehicles using pre-training and fine-tuning on real data.
• Uses a curriculum-style training setup where policies are first trained in simulation and then adapted on real vehicle data.
• Evaluates performance using success rates on training and test sets, with metrics tracking learning speed and generalization.

Experimental results

Research questions

• RQ1 How well does a DQN trained on one intersection type generalize to another when directly transferred (direct copy)?
• RQ2 Does fine-tuning a pre-trained DQN on a new intersection type lead to faster convergence and better asymptotic performance than random initialization?
• RQ3 To what extent is knowledge of the original task retained after fine-tuning on a new task (reverse transfer)?
• RQ4 Can a policy pre-trained in simulation be effectively fine-tuned on real-world data to improve real vehicle performance?
• RQ5 How does the performance of a sim-to-real transfer policy compare to a policy trained solely on real data in terms of training speed and generalization?

Key findings

• Direct transfer of a DQN from one intersection type to another consistently resulted in low success rates, indicating poor generalization.
• Fine-tuning a pre-trained DQN on a new intersection type led to faster learning and higher asymptotic performance than random initialization in nearly all cases.
• Reverse transfer showed that knowledge of the source task was partially retained after fine-tuning, with performance on the source task outperforming direct copy but underperforming a network trained from scratch.
• Sim-to-real transfer significantly accelerated training: a sim-pretrained network reached 90% success on training data in half the iterations required by a real-only trained network.
• Despite faster convergence, both real-only and sim-to-real fine-tuned networks exhibited overfitting, with test performance asymptoting at around 80% success.
• Transfer from real data to simulation was less effective, with fine-tuned real policies failing to generalize to simulation, likely due to overfitting on limited real data.