[Paper Review] DisCoRL: Continual Reinforcement Learning via Policy Distillation

Kalifou René Traoré, Hugo Caselles-Dupré+5 more|arXiv (Cornell University)|Jul 11, 2019

Domain Adaptation and Few-Shot Learning21 references35 citations

TL;DR

DisCoRL combines state representation learning and policy distillation to perform continual reinforcement learning across sequential tasks without task labels at test time, evaluated in simulation and transferred to real life.

ABSTRACT

In multi-task reinforcement learning there are two main challenges: at training time, the ability to learn different policies with a single model; at test time, inferring which of those policies applying without an external signal. In the case of continual reinforcement learning a third challenge arises: learning tasks sequentially without forgetting the previous ones. In this paper, we tackle these challenges by proposing DisCoRL, an approach combining state representation learning and policy distillation. We experiment on a sequence of three simulated 2D navigation tasks with a 3 wheel omni-directional robot. Moreover, we tested our approach's robustness by transferring the final policy into a real life setting. The policy can solve all tasks and automatically infer which one to run.

Motivation & Objective

• Motivate autonomous agents to learn skills sequentially without forgetting and without relying on task labels during testing.
• Develop a pipeline that merges state representation learning with policy distillation for continual RL.
• Demonstrate that distilled policies can solve multiple tasks and transfer to real-world robotic settings.

Proposed method

• Learn a task-specific state representation encoder E_i using SRL from data collected with a random policy on Env_i.
• Train a policy pi_i in the learned state space using RL.
• Create a distillation dataset D_pi_i by recording observations and action probabilities from pi_i along trajectories.
• Distill pi_i into a shared policy pi_d:(1..i) by combining D_pi_i with past distillation datasets and training a student to imitate teachers.
• Aggregate distilled policies into a single policy pi_d:1..n that can select appropriate actions without a task indicator.
• Evaluate final distilled policy in simulation and transfer to real life using domain randomization and SRL.

Experimental results

Research questions

• RQ1 Can a single distilled policy run multiple sequential RL tasks without task labels at test time?
• RQ2 Does combining SRL with policy distillation prevent forgetting across tasks in a continual RL setting?
• RQ3 How well does the distilled policy transfer from simulation to a real robot with domain randomization?

Key findings

• The pipeline learns continually three sequential 2D navigation tasks in simulation and transfers the final policy to a real robot.
• Policy distillation memory uses only data (D_pi_i) rather than saving all teacher models, enabling scalable continual RL.
• Each task’s RL policy pi_i is learned efficiently and then distilled into a single policy with modest computational cost relative to policy learning.
• The method does not require task labels at test time, relying on visual cues to infer the current task.
• Experiments include domain randomization to aid sim-to-real transfer and show robustness to the reality gap.
• The distillation dataset per task contains approximately 10k samples, enabling faster distillation compared to full RL training.