[Paper Review] Interesting Object, Curious Agent: Learning Task-Agnostic Exploration

Simone Parisi, Victoria Dean+2 more|arXiv (Cornell University)|Nov 25, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Change-Based Exploration Transfer (C-BET), a task-agnostic exploration framework that learns joint agent-centric and environment-centric exploration policies across multiple environments for improved transfer to unseen environments. C-BET outperforms prior methods by combining intrinsic curiosity with environment-level interestingness, achieving superior coverage and consistency in diverse testing scenarios.

ABSTRACT

Common approaches for task-agnostic exploration learn tabula-rasa --the agent assumes isolated environments and no prior knowledge or experience. However, in the real world, agents learn in many environments and always come with prior experiences as they explore new ones. Exploration is a lifelong process. In this paper, we propose a paradigm change in the formulation and evaluation of task-agnostic exploration. In this setup, the agent first learns to explore across many environments without any extrinsic goal in a task-agnostic manner. Later on, the agent effectively transfers the learned exploration policy to better explore new environments when solving tasks. In this context, we evaluate several baseline exploration strategies and present a simple yet effective approach to learning task-agnostic exploration policies. Our key idea is that there are two components of exploration: (1) an agent-centric component encouraging exploration of unseen parts of the environment based on an agent's belief; (2) an environment-centric component encouraging exploration of inherently interesting objects. We show that our formulation is effective and provides the most consistent exploration across several training-testing environment pairs. We also introduce benchmarks and metrics for evaluating task-agnostic exploration strategies. The source code is available at https://github.com/sparisi/cbet/.

Motivation & Objective

• To address the limitation of tabula-rasa exploration in reinforcement learning by modeling exploration as a lifelong, transferable process.
• To introduce a paradigm shift from isolated, single-environment exploration to multi-environment pre-training with transfer to new environments.
• To identify and model two components of exploration: agent-centric (unseen states) and environment-centric (inherently interesting objects).
• To develop benchmarks and metrics for evaluating task-agnostic exploration strategies in transfer settings.
• To demonstrate that learning from diverse environments improves generalization and transferability of exploration policies.

Proposed method

• C-BET learns exploration policies using a combination of agent-centric surprise (based on state visitation counts) and environment-centric change detection (using panoramic or egocentric visual changes).
• The method uses a dual-component intrinsic reward: one based on the agent’s belief about unexplored states, and another based on high-impact environmental changes (e.g., objects that trigger interactions like doors or switches).
• Pre-training occurs across multiple environments without extrinsic rewards, enabling the agent to learn transferable exploration behaviors.
• During transfer, the pre-trained policy is applied to new, unseen environments with no further training, evaluating coverage and consistency.
• The framework uses both panoramic and egocentric visual encoding to assess environmental changes, with ablation studies showing the importance of combining both modalities.
• Ablation studies compare C-BET against baselines like Count, Curiosity, RND, and RIDE, evaluating policy distributions and robustness under noisy conditions.

Experimental results

Research questions

• RQ1 Can exploration policies pre-trained across multiple environments generalize effectively to unseen environments without task-specific rewards?
• RQ2 How does combining agent-centric and environment-centric exploration components improve transfer performance compared to agent-centric methods alone?
• RQ3 What role do environmental objects with high intrinsic interestingness (e.g., switches, keys) play in guiding effective exploration?
• RQ4 How does noise in the environment affect the robustness and transferability of exploration policies?
• RQ5 To what extent does pre-training on diverse environments reduce overfitting to specific actions or states?

Key findings

• C-BET achieves the highest and most consistent scene coverage across all unseen environments, visiting nearly all states in smaller scenes like Room 0 and Room 1, and most of larger scenes like Apt. 0 and Hotel 0 within a single episode.
• C-BET outperforms all baselines (Count, Curiosity, RND, RIDE) in both pre-training and transfer phases, especially in environments with high visual diversity.
• Even with egocentric visual encoding, C-BET maintains strong performance, demonstrating that its success stems from combining egocentric state counts with panoramic change detection, not just panoramic counts.
• In noisy environments, C-BET’s policy distribution remains stable, assigning low probability to 'drop' and maintaining consistent action probabilities, unlike Count, which overfits to 'forward' and underperforms.
• The presence of resets during pre-training is crucial for Count’s performance, but C-BET maintains strong transferability even without resets, showing greater robustness.
• C-BET’s policy entropy remains low (0.74–0.99) across setups, indicating stable and focused behavior, while baselines with no resets become nearly random, especially under noise.