[Paper Review] Federated Transfer Reinforcement Learning for Autonomous Driving

Xinle Liang, Yang Liu+3 more|arXiv (Cornell University)|Oct 14, 2019

Autonomous Vehicle Technology and SafetyEngineering38 citations

TL;DR

This paper presents an online federated reinforcement learning transfer framework (FTRL) that enables real-time knowledge sharing among heterogeneous agents in simulators and real-world environments for autonomous driving, validated with AirSim and JetsonTX2 RC cars showing improved collision avoidance performance.

ABSTRACT

Reinforcement learning (RL) is widely used in autonomous driving tasks and training RL models typically involves in a multi-step process: pre-training RL models on simulators, uploading the pre-trained model to real-life robots, and fine-tuning the weight parameters on robot vehicles. This sequential process is extremely time-consuming and more importantly, knowledge from the fine-tuned model stays local and can not be re-used or leveraged collaboratively. To tackle this problem, we present an online federated RL transfer process for real-time knowledge extraction where all the participant agents make corresponding actions with the knowledge learned by others, even when they are acting in very different environments. To validate the effectiveness of the proposed approach, we constructed a real-life collision avoidance system with Microsoft Airsim simulator and NVIDIA JetsonTX2 car agents, which cooperatively learn from scratch to avoid collisions in indoor environment with obstacle objects. We demonstrate that with the proposed framework, the simulator car agents can transfer knowledge to the RC cars in real-time, with 27% increase in the average distance with obstacles and 42% decrease in the collision counts.

Motivation & Objective

• Address the time-consuming offline model transfer in autonomous driving RL workflows and overcome local knowledge silos.
• Propose an online federated RL transfer framework (FTRL) that supports asynchronous updates and cross-environment knowledge sharing.
• Validate FTRL on real JetsonTX2 RC cars and Microsoft AirSim simulator for collision avoidance tasks.
• Demonstrate that online transfer and federation improve learning speed and evaluation metrics compared to non-federated local training.

Proposed method

• Combine federated learning (FedAvg) with transfer learning to enable asynchronous RL updates across agents in different environments.
• Use Deep Deterministic Policy Gradient (DDPG) as the underlying RL method with an online transfer mechanism for aligning observations and actions across environments (s_t = beta_i s_t^i; a_t^i = a_t * |Max a^i|).
• Federation model w_fed^θ is updated as the average of local models (w_fed^θ = (1/N) Σ_i w_i^θ).
• Define and apply a reward function conditioned on the next state r_t(s_{t+1}) with specific terms for safe distance and obstacle penalties (Eq. 2).
• Implement asynchronous training with local update interval t_u and federation interval t_f; server aggregates models and distributes federation model.
• Describe DDPG network architecture (actor and critic with 3 fully-connected layers of 128 units) and standard RL settings (γ=0.99, τ=0.02, lr=1e-4, replay buffer=2500, batch=32).
• Detail the transfer mechanism for heterogeneous sensors (LIDAR) and non-identical environments via observation and action scaling (β_i, Max action normalization).

Experimental results

Research questions

• RQ1 Can online transfer and federation enable knowledge sharing between non-identical RL tasks/environments (simulator vs real robots)?
• RQ2 Does FTRL improve training speed and evaluation performance compared to a single non-federated RL training run?
• RQ3 Is online simulator-to-real knowledge transfer feasible and beneficial for autonomous driving collision avoidance tasks?

Key findings

• FTRL and FTRL-SIM demonstrate higher relative performance than baseline DDPG in tested scenarios.
• Compared with DDPG, FTRL achieves average distance improvements and fewer collisions across three RC cars: car1, car2, car3.
• FTRL-DDPG-SIM further improves performance by transferring knowledge from AirSim simulator to real cars, yielding larger relative gains.
• Table I shows per-car results: DDPG (car1: avg_dist 0.39, coll_no 18; car2: 0.29, 31; car3: 0.38, 24); FTRL-DDPG (car1: 0.42, 9; car2: 0.37, 27; car3: 0.51, 17); FTRL-DDPG-SIM (car1: 0.45, 12; car2: 0.39, 16; car3: 0.50, 13).
• Overall improvements reported: average 20.3% increase in average distance with obstacles and 30.7% decrease in collisions for FTRL-DDPG; 27.2% distance and 42.5% collisions for FTRL-DDPG-SIM when averaged over test races.