[Paper Review] Augmenting Autotelic Agents with Large Language Models

Cédric Colas, Laetitia Teodorescu+3 more|arXiv (Cornell University)|May 21, 2023

Topic Modeling4 citations

TL;DR

This paper introduces LMA3, a language model-augmented autotelic agent that uses a pretrained language model to generate, relabel, and reward diverse, abstract, and human-relevant goals in a task-agnostic text-based environment. Without hand-coded goals or rewards, LMA3 achieves open-ended skill learning by leveraging the LM for goal generation, trajectory relabeling, and reward shaping, demonstrating mastery of a wide range of skills through self-generated, creative objectives.

ABSTRACT

Humans learn to master open-ended repertoires of skills by imagining and practicing their own goals. This autotelic learning process, literally the pursuit of self-generated (auto) goals (telos), becomes more and more open-ended as the goals become more diverse, abstract and creative. The resulting exploration of the space of possible skills is supported by an inter-individual exploration: goal representations are culturally evolved and transmitted across individuals, in particular using language. Current artificial agents mostly rely on predefined goal representations corresponding to goal spaces that are either bounded (e.g. list of instructions), or unbounded (e.g. the space of possible visual inputs) but are rarely endowed with the ability to reshape their goal representations, to form new abstractions or to imagine creative goals. In this paper, we introduce a language model augmented autotelic agent (LMA3) that leverages a pretrained language model (LM) to support the representation, generation and learning of diverse, abstract, human-relevant goals. The LM is used as an imperfect model of human cultural transmission; an attempt to capture aspects of humans' common-sense, intuitive physics and overall interests. Specifically, it supports three key components of the autotelic architecture: 1)~a relabeler that describes the goals achieved in the agent's trajectories, 2)~a goal generator that suggests new high-level goals along with their decomposition into subgoals the agent already masters, and 3)~reward functions for each of these goals. Without relying on any hand-coded goal representations, reward functions or curriculum, we show that LMA3 agents learn to master a large diversity of skills in a task-agnostic text-based environment.

Motivation & Objective

• To enable autotelic agents to learn open-ended skills by generating and pursuing self-invented, abstract, and creative goals.
• To overcome the limitations of current artificial agents that rely on fixed or predefined goal spaces.
• To leverage large language models as a proxy for cultural transmission, enabling the evolution of goal representations through language.
• To develop a framework for autonomous goal generation, relabeling, and reward shaping without human-designed reward functions or curricula.
• To demonstrate that language models can support scalable, diverse, and human-relevant skill acquisition in task-agnostic environments.

Proposed method

• The LMA3 framework integrates a pretrained language model (LM) as a cultural transmission mechanism to support three core components: goal generation, trajectory relabeling, and reward function creation.
• The goal generator prompts the LM with prior trajectories and mastered subgoals to produce high-level, abstract goals and their hierarchical decomposition into executable subgoals.
• The relabeler uses the LM to re-describe agent trajectories in natural language, identifying novel or complex goals achieved during rollouts.
• Reward functions are automatically generated by the LM based on the relabeled goal descriptions, enabling intrinsic motivation for skill acquisition.
• The agent performs rollouts using a learned policy to execute subgoal sequences, with success evaluated via LM-based relabeling and reward shaping.
• The system operates in a task-agnostic text-based environment, relying solely on language for goal representation and feedback, avoiding hand-coded objectives or reward functions.

Experimental results

Research questions

• RQ1 Can a large language model effectively generate diverse, abstract, and human-relevant goals without human-provided goal templates?
• RQ2 Can language model-based relabeling and reward shaping enable open-ended skill learning in a task-agnostic environment?
• RQ3 How does the integration of language models into autotelic agents affect the diversity and complexity of learned skills?
• RQ4 To what extent can language models serve as a proxy for cultural transmission in shaping goal representations across agents?
• RQ5 Can the agent achieve meaningful skill acquisition without predefined rewards, curricula, or hand-coded goal spaces?

Key findings

• LMA3 agents successfully learn to master a wide diversity of skills in a task-agnostic text-based environment without any hand-coded goal representations or reward functions.
• The language model effectively generates high-level, abstract, and creative goals such as 'prepare cilantro dish' and 'enjoy delicious meal' based on agent trajectories and mastered subgoals.
• The relabeling component of the LM successfully identifies and re-describes complex or novel goals achieved during rollouts, such as 'open fridge and then close it' or 'eat healthy'.
• The system demonstrates open-ended learning by recursively composing mastered subgoals into increasingly complex goals, enabling the agent to explore a broad space of possible skills.
• The approach achieves skill acquisition with minimal human intervention, relying solely on language model interactions and a learned policy, with 550k LM calls used in experiments.
• The framework shows promise for scalable, human-relevant skill learning, though high inference costs (≈$240 per 10k episodes) remain a practical limitation.