[论文解读] A Definition of Continual Reinforcement Learning

David Abel, André Barreto+4 more|arXiv (Cornell University)|Jul 20, 2023

Reinforcement Learning in Robotics被引用 9

一句话总结

本论文通过将智能体形式化为对基础智能体集合的持续隐式搜索来定义持续强化学习（CRL），并证明最优智能体永不停止学习。同时将CRL与多任务强化学习和持续监督学习等情形联系起来。

ABSTRACT

In a standard view of the reinforcement learning problem, an agent's goal is to efficiently identify a policy that maximizes long-term reward. However, this perspective is based on a restricted view of learning as finding a solution, rather than treating learning as endless adaptation. In contrast, continual reinforcement learning refers to the setting in which the best agents never stop learning. Despite the importance of continual reinforcement learning, the community lacks a simple definition of the problem that highlights its commitments and makes its primary concepts precise and clear. To this end, this paper is dedicated to carefully defining the continual reinforcement learning problem. We formalize the notion of agents that "never stop learning" through a new mathematical language for analyzing and cataloging agents. Using this new language, we define a continual learning agent as one that can be understood as carrying out an implicit search process indefinitely, and continual reinforcement learning as the setting in which the best agents are all continual learning agents. We provide two motivating examples, illustrating that traditional views of multi-task reinforcement learning and continual supervised learning are special cases of our definition. Collectively, these definitions and perspectives formalize many intuitive concepts at the heart of learning, and open new research pathways surrounding continual learning agents.

研究动机与目标

• 将持续强化学习的研究动机从解决单一问题转向无休止的适应性.
• 使用智能体基础及两个核心运算符（生成（generates）和到达（reaches））给出CRL的正式、数学定义。
• 证明传统的多任务强化学习和持续监督学习是CRL的特例。
• 指明CRL及其运算符的必要特性，以指导未来研究。

提出的方法

• 引入具有历史、环境和基于奖励的性能的通用智能体-环境框架。
• 定义智能体基础与学习规则；形式化智能体如何通过生成运算符（Definition 3.3, 3.4）生成其他智能体。
• 定义到达运算符（有时到达、有时不到达）以刻画智能体是否最终稳定在某个基础上（Definitions 3.5–3.6）。
• 证明 Theorem 3.1：任何智能体都可被视为在基础上隐式搜索；Remark 3.2：智能体要么有时到达要么从不到达基础。
• 将持续学习智能体定义为在生成基础的同时从不到达基础的智能体（Definitions 4.1–4.2）。
• 给出CRL示例（切换MDP和持续监督学习）以说明该框架。

实验结果

研究问题

• RQ1 如何正式定义持续强化学习以捕捉持续的适应性？
• RQ2 智能体是否可被描述为在有限或无限的基础集合上进行隐式搜索？
• RQ3 在何种条件下最佳智能体永不停止搜索，使CRL成为自然的设定？
• RQ4 CRL概念与现有观点（如多任务RL和持续监督学习）之间的关系为何？

主要发现

• CRL通过两个运算符来定义：生成（generates，智能体从基础集合形成）和到达（reaches，智能体是否在基础上定居）。
• 任何智能体都可被解释为在某个智能体基础上进行隐式搜索（Theorem 3.1）。
• 智能体要么有时到达要么从不到达其基础（Remark 3.2）。
• CRL在最佳智能体永不停歇地对基础进行隐式搜索时发生（Definition 4.2）。
• CRL示例表明，在非平稳切换下的收敛Q学习被持续学习超越，表明需要不断在基础上进行搜索。