[Paper Review] A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents

Olivier Sigaud, Gianluca Baldassarre+6 more|arXiv (Cornell University)|Nov 1, 2023

Reinforcement Learning in Robotics58 references4 citations

TL;DR

This paper proposes a formal definition of open-ended learning (OEL) for goal-conditioned agents by isolating a core property: the continuous, infinite-horizon generation of novel goals, options, or reward functions from an observer's perspective. It introduces open-ended goal-conditioned reinforcement learning (GCRL) problems and establishes a framework to evaluate agents based on their capacity to discover and master new goals over time, distinguishing OEL from related concepts like continual or lifelong learning.

ABSTRACT

A lot of recent machine learning research papers have ``open-ended learning'' in their title. But very few of them attempt to define what they mean when using the term. Even worse, when looking more closely there seems to be no consensus on what distinguishes open-ended learning from related concepts such as continual learning, lifelong learning or autotelic learning. In this paper, we contribute to fixing this situation. After illustrating the genealogy of the concept and more recent perspectives about what it truly means, we outline that open-ended learning is generally conceived as a composite notion encompassing a set of diverse properties. In contrast with previous approaches, we propose to isolate a key elementary property of open-ended processes, which is to produce elements from time to time (e.g., observations, options, reward functions, and goals), over an infinite horizon, that are considered novel from an observer's perspective. From there, we build the notion of open-ended learning problems and focus in particular on the subset of open-ended goal-conditioned reinforcement learning problems in which agents can learn a growing repertoire of goal-driven skills. Finally, we highlight the work that remains to be performed to fill the gap between our elementary definition and the more involved notions of open-ended learning that developmental AI researchers may have in mind.

Motivation & Objective

• To address the lack of consensus and formal definition in the growing literature on open-ended learning (OEL) in machine learning.
• To isolate a core, elementary property of OEL: the generation of novel elements (e.g., goals, options) over an infinite time horizon from an observer’s viewpoint.
• To define open-ended learning problems specifically within the context of goal-conditioned reinforcement learning (GCRL).
• To distinguish OEL from related concepts such as continual learning, lifelong learning, and autotelic learning through a principled framework.
• To identify open research directions for capturing the developmental trajectory of skill acquisition, including representational redescription and abstraction, in OEL agents.

Proposed method

• The paper defines OEL through a core property: the agent generates novel elements (e.g., goals, options, reward functions) over an infinite time horizon, where novelty is assessed from an external observer’s perspective.
• It introduces open-ended GCRL problems where agents learn in two phases: an intrinsic phase with no external guidance, followed by an extrinsic phase with randomly drawn tasks from the environment’s task space.
• Performance in the extrinsic phase is used as a proxy for knowledge-gain capacity during the intrinsic phase, measuring how well the agent generalizes to unseen tasks.
• The paper proposes evaluating OEL agents by tracking the rate of novel goal discovery over time, with asymptotic convergence indicating non-OEL behavior.
• It suggests using a common task set in the extrinsic phase to fairly compare agents with different intrinsic curricula, enabling performance-based comparison across OEL agents.
• The framework emphasizes the need to combine the core novelty property with additional capabilities—such as skill abstraction, representational redescription, and creativity—to reflect developmental learning in humans.

Experimental results

Research questions

• RQ1 What is the fundamental, elementary property that defines open-ended learning in autonomous agents?
• RQ2 How can open-ended learning problems be formally defined within goal-conditioned reinforcement learning?
• RQ3 In what way does open-ended learning differ from related concepts such as continual learning, lifelong learning, or autotelic learning?
• RQ4 How can the performance of OEL agents be fairly evaluated when they follow different intrinsic curricula?
• RQ5 What additional capabilities—beyond novelty generation—are necessary to capture the developmental trajectory of skill acquisition in OEL agents?

Key findings

• The core property of open-ended learning is the continuous, infinite-horizon generation of novel elements (e.g., goals, options, reward functions) from an observer’s perspective.
• The paper establishes a formal framework for open-ended goal-conditioned reinforcement learning (GCRL) problems, distinguishing them from standard RL settings.
• Agents that fail to discover new goals over time—evidenced by a converging number of seen goals—do not qualify as open-ended learning agents.
• A logarithmic or linearly increasing rate of novel goal discovery over time indicates open-ended learning behavior, with increasing slope being the strongest indicator.
• Performance in a standardized extrinsic phase with randomly drawn tasks serves as a valid proxy for measuring the knowledge-gain capacity of OEL processes in the intrinsic phase.
• The framework highlights the need to integrate additional developmental mechanisms—such as representational redescription, abstraction, and creativity—into OEL agents to reflect human-like skill acquisition.