[Paper Review] Transfer Learning in Deep Reinforcement Learning: A Survey

Zhuangdi Zhu, Kaixiang Lin+2 more|arXiv (Cornell University)|Sep 16, 2020

Reinforcement Learning in Robotics178 references150 citations

TL;DR

A comprehensive survey of how transfer learning is applied to deep reinforcement learning, detailing knowledge transfer types, taxonomy, evaluation metrics, and future directions.

ABSTRACT

Reinforcement learning is a learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in reinforcement learning upon the fast development of deep neural networks. Along with the promising prospects of reinforcement learning in numerous domains such as robotics and game-playing, transfer learning has arisen to tackle various challenges faced by reinforcement learning, by transferring knowledge from external expertise to facilitate the efficiency and effectiveness of the learning process. In this survey, we systematically investigate the recent progress of transfer learning approaches in the context of deep reinforcement learning. Specifically, we provide a framework for categorizing the state-of-the-art transfer learning approaches, under which we analyze their goals, methodologies, compatible reinforcement learning backbones, and practical applications. We also draw connections between transfer learning and other relevant topics from the reinforcement learning perspective and explore their potential challenges that await future research progress.

Motivation & Objective

• Define transfer learning in the context of reinforcement learning and DRL.
• Systematically categorize TL approaches by what knowledge is transferred and how it is used across DRL backbones.
• Analyze the goals, methodologies, and applications of TL methods in DRL.
• Discuss evaluation metrics and future research directions for TL in DRL.

Proposed method

• Propose a framework to categorize TL approaches in DRL based on the form of transferred knowledge and the transfer process.
• Review TL methods organized by the format of transferred knowledge (e.g., reward shaping, learning from demonstrations, teacher policies, representations).
• Analyze compatibility with RL backbones, and differences between source and target domains.
• Summarize evaluation metrics for TL in DRL and discuss new metrics related to knowledge quality and quantity.

Experimental results

Research questions

• RQ1 What forms of knowledge can be transferred in DRL to facilitate learning?
• RQ2 How do different TL approaches align with various DRL backbones and task differences?
• RQ3 What metrics best evaluate TL effectiveness and the quality of transferred knowledge in DRL?
• RQ4 What are the future directions and open challenges for TL in DRL?

Key findings

• Reward shaping, demonstration learning, and policy transfer are core TL approaches in DRL with different compatibility to backbones.
• PBRS, PBA, DPB, and DPBA provide a spectrum of potential-based reward shaping methods for TL in DRL.
• Learning from demonstrations and teacher policies enable policy-agnostic and policy-distillation forms of knowledge transfer across DRL tasks.
• TL evaluation considers both mastery (final performance) and generalization (speed and robustness), plus proposed knowledge-centric metrics like necessary knowledge amount and quality.
• The survey identifies future directions such as reasoning over diverse knowledge forms and efficient, principled knowledge usage for TL in DRL.