[Paper Review] Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

Sanmit Narvekar, Bei Peng+4 more|arXiv (Cornell University)|Mar 10, 2020

Reinforcement Learning in Robotics114 references228 citations

TL;DR

The paper proposes a formal framework for curriculum learning in reinforcement learning, surveys existing methods, and identifies open problems and directions for future research.

ABSTRACT

Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. Despite many advances over the past three decades, learning in many domains still requires a large amount of interaction with the environment, which can be prohibitively expensive in realistic scenarios. To address this problem, transfer learning has been applied to reinforcement learning such that experience gained in one task can be leveraged when starting to learn the next, harder task. More recently, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum for the purpose of learning a problem that may otherwise be too difficult to learn from scratch. In this article, we present a framework for curriculum learning (CL) in reinforcement learning, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals. Finally, we use our framework to find open problems and suggest directions for future RL curriculum learning research.

Motivation & Objective

• Define a general and formal notion of curriculum for RL tasks and samples.
• Classify curriculum learning approaches by representation, transfer, adaptivity, and evaluation metrics.
• Survey existing methods for task generation, sequencing, and transfer in RL curricula.
• Identify gaps, open problems, and future directions in RL curriculum learning.

Proposed method

• Formalize Curriculum as a directed acyclic graph over task samples or tasks with a mapping function g to sample sets.
• Distinguish single-task, task-level, and sequence curricula to simplify practical use.
• Extend transfer learning evaluation metrics to the curriculum setting, including weak vs strong transfer considerations.
• Categorize curriculum approaches along seven dimensions including task generation, representation, transfer method, sequencer, adaptivity, evaluation, and application domain.
• Survey task generation, sequencing algorithms (with emphasis on sequencing), and transfer learning methods used within RL curricula.
• Provide a framework to compare curricula by stopping criteria and cost considerations for curriculum design.

Experimental results

Research questions

• RQ1 What constitutes a curriculum in reinforcement learning, and how can it be represented for RL tasks?
• RQ2 How can curricula be generated, sequenced, and evaluated within RL settings?
• RQ3 What transfer mechanisms are used across tasks in a curriculum, and how do they influence learning?
• RQ4 How should curricula be categorized and what open problems remain in RL curriculum learning?

Key findings

• A general curriculum framework is proposed, allowing representation as graphs of samples or tasks with a sink node.
• Curricula can be static or adaptive, and evaluation should account for curriculum design cost in addition to target-task performance.
• Most existing work focuses on sequencing, with task generation and transfer methods also surveyed and categorized.
• Seven dimensions are proposed for categorization, enabling systematic taxonomy across the literature.
• Curriculum generation often relies on human-designed curricula, with automated methods increasingly explored.
• Open problems and future directions are identified for RL curriculum learning.