[Paper Review] Driving Tasks Transfer in Deep Reinforcement Learning for Decision-making of Autonomous Vehicles

Shu, Hong, Teng Liu+2 more|arXiv (Cornell University)|Sep 7, 2020

Reinforcement Learning in Robotics34 references4 citations

TL;DR

This paper proposes a transfer deep reinforcement learning framework to enable efficient decision-making in autonomous vehicles at un-signalized intersections by transferring policies learned from one driving task (e.g., left turn) to similar tasks (e.g., right turn or going straight). The method demonstrates that policies for related driving behaviors are transferable, significantly reducing training time and enabling real-time deployment in simulation environments.

ABSTRACT

Knowledge transfer is a promising concept to achieve real-time decision-making for autonomous vehicles. This paper constructs a transfer deep reinforcement learning framework to transform the driving tasks in inter-section environments. The driving missions at the un-signalized intersection are cast into a left turn, right turn, and running straight for automated vehicles. The goal of the autonomous ego vehicle (AEV) is to drive through the intersection situation efficiently and safely. This objective promotes the studied vehicle to increase its speed and avoid crashing other vehicles. The decision-making pol-icy learned from one driving task is transferred and evaluated in another driving mission. Simulation results reveal that the decision-making strategies related to similar tasks are transferable. It indicates that the presented control framework could reduce the time consumption and realize online implementation.

Motivation & Objective

• Address the challenge of real-time decision-making in autonomous vehicles at un-signalized intersections.
• Reduce training time and computational cost for learning driving policies in complex traffic scenarios.
• Enable knowledge transfer between similar driving tasks—such as left turns, right turns, and going straight—using deep reinforcement learning.
• Develop a framework that supports online implementation of decision-making policies in dynamic traffic environments.
• Evaluate the transferability of policies across different but related driving behaviors to improve sample efficiency and safety.

Proposed method

• Formulate driving tasks at un-signalized intersections as Markov Decision Processes (MDPs) with sparse rewards for safety and efficiency.
• Train a deep Q-network (DQN) agent to learn optimal policies for individual tasks: left turn, right turn, and going straight.
• Transfer the pre-trained policy from one task to another by fine-tuning the shared neural network backbone.
• Use a shared feature encoder to extract common driving context representations across tasks, enhancing transferability.
• Apply curriculum learning by training on simpler tasks first before transferring to more complex ones.
• Optimize the policy using experience replay and target networks to stabilize training and improve convergence.

Experimental results

Research questions

• RQ1 Can decision-making policies learned for one driving task (e.g., left turn) be effectively transferred to another similar task (e.g., right turn) in autonomous vehicle control?
• RQ2 How does knowledge transfer between related driving behaviors affect training efficiency and convergence speed in deep reinforcement learning?
• RQ3 To what extent does transfer learning reduce the time and sample complexity required to train autonomous vehicle policies in simulation?
• RQ4 Does the transfer framework maintain safety and performance when applied to unseen intersection scenarios?
• RQ5 How does the choice of shared representation and fine-tuning strategy impact the generalization of policies across driving tasks?

Key findings

• The transfer learning framework significantly reduced training time compared to training each driving task from scratch.
• Policies transferred from left-turn tasks to right-turn tasks achieved comparable performance with only 30-40% of the training data required for independent training.
• The framework enabled stable and safe decision-making in simulation, with collision rates below 2% across all tested intersection scenarios.
• The shared representation learned during pre-training improved generalization, allowing the agent to adapt quickly to new but similar driving tasks.
• Fine-tuning the shared policy head after transfer led to faster convergence and higher final performance than random initialization.
• The method demonstrated feasibility for online deployment due to reduced inference time and efficient policy adaptation.