[Paper Review] A Definition of Continual Reinforcement Learning

David Abel, André Barreto+4 more|arXiv (Cornell University)|Jul 20, 2023

Reinforcement Learning in Robotics9 citations

TL;DR

This paper defines continual reinforcement learning (CRL) by formalizing agents as ongoing implicit searches over a basis of base agents, and shows that the best agents never stop learning. It also connects CRL to multi-task RL and continual supervised learning as special cases.

ABSTRACT

In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution, rather than treating learning as endless adaptation. In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.

Motivation & Objective

• Motivate the study of continual reinforcement learning as a departure from solving problems to endless adaptation.
• Provide a formal, mathematical definition of CRL using agent bases and two core operators (generates and reaches).
• Show that traditional multi-task RL and continual supervised learning are special cases of CRL.
• Identify necessary properties of CRL and its operators to guide future research.

Proposed method

• Introduce a general agent-environment framework with histories, environments, and reward-based performance.
• Define agent basis and learning rules; formalize how an agent can generate other agents via the generates operator (Definition 3.3, 3.4).
• Define the reaches operators (sometimes reaches and never reaches) to capture whether an agent settles on a basis (Definitions 3.5–3.6).
• Prove Theorem 3.1: any agent can be viewed as implicitly searching over a basis; Remark 3.2: an agent either sometimes or never reaches the basis.
• Define continual learning agents as those that generate the basis while never reaching it (Definitions 4.1–4.2).
• Provide CRL examples (switching MDPs and continual supervised learning) to illustrate the framework.

Experimental results

Research questions

• RQ1 How can continual reinforcement learning be formally defined to capture ongoing adaptation?
• RQ2 Can agents be characterized as implicit searches over a finite or infinite basis of base agents?
• RQ3 Under what conditions do best agents never stop searching, making CRL the natural setting?
• RQ4 How do CRL concepts relate to existing views like multi-task RL and continual supervised learning?

Key findings

• CRL is defined via two operators: generates (agents form from a basis) and reaches (whether an agent settles on a basis).
• Any agent can be interpreted as implicitly searching over some agent basis (Theorem 3.1).
• An agent will either sometimes reach or never reach its basis (Remark 3.2).
• CRL occurs when the best agents never stop the implicit search over the basis (Definition 4.2).
• CRL examples show that convergent Q-learning under non-stationary switches is outperformed by continual learning, illustrating the need to keep searching over the basis.