[Paper Review] BeBold: Exploration Beyond the Boundary of Explored Regions

Tianjun Zhang, Huazhe Xu+5 more|arXiv (Cornell University)|Dec 15, 2020

Reinforcement Learning in RoboticsComputer Science55 references18 citations

TL;DR

BeBold proposes a novel intrinsic reward (IR) mechanism that uses the regulated difference of inverse visitation counts between consecutive states to drive exploration beyond explored regions, mitigating short-sightedness and detachment in count-based methods. It achieves state-of-the-art performance by solving all 12 most challenging procedurally-generated MiniGrid tasks within 120M environment steps without curriculum learning, outperforming prior methods.

ABSTRACT

Efficient exploration under sparse rewards remains a key challenge in deep reinforcement learning. To guide exploration, previous work makes extensive use of intrinsic reward (IR). There are many heuristics for IR, including visitation counts, curiosity, and state-difference. In this paper, we analyze the pros and cons of each method and propose the regulated difference of inverse visitation counts as a simple but effective criterion for IR. The criterion helps the agent explore Beyond the Boundary of explored regions and mitigates common issues in count-based methods, such as short-sightedness and detachment. The resulting method, BeBold, solves the 12 most challenging procedurally-generated tasks in MiniGrid with just 120M environment steps, without any curriculum learning. In comparison, the previous SoTA only solves 50% of the tasks. BeBold also achieves SoTA on multiple tasks in NetHack, a popular rogue-like game that contains more challenging procedurally-generated environments.

Motivation & Objective

• To address the challenge of efficient exploration under sparse rewards in deep reinforcement learning.
• To overcome limitations in existing intrinsic reward methods, such as short-sightedness, detachment, and asymptotic inconsistency in state-difference approaches.
• To develop a simple yet effective exploration criterion that encourages agents to push beyond the boundary of explored regions.
• To achieve state-of-the-art performance on challenging procedural environments like MiniGrid and NetHack without curriculum learning or complex curriculum design.

Proposed method

• Proposes a new intrinsic reward based on the regulated difference of inverse visitation counts between consecutive states in a trajectory.
• Uses Random Network Distillation (RND) to approximate inverse visitation counts, enabling efficient and scalable estimation.
• Defines intrinsic reward as max(1/N(s_{t+1}) - 1/N(s_t), 0), where N(s) is the visitation count of state s, ensuring high reward at the frontier of exploration.
• Introduces a 'regulated' difference to prevent spurious rewards from stochastic environment dynamics, improving stability.
• Employs a hash table to maintain episodic visitation counts for efficient on-policy tracking.
• Applies the method in a standard deep RL framework with shared architecture and hyperparameters across environments.

Experimental results

Research questions

• RQ1 Can a simple, count-based intrinsic reward mechanism effectively drive exploration beyond the boundary of explored regions?
• RQ2 How does the regulated difference of inverse visitation counts compare to existing IR methods in mitigating short-sightedness and detachment?
• RQ3 Can this method achieve state-of-the-art performance on challenging procedural environments like MiniGrid and NetHack without curriculum learning?
• RQ4 To what extent does the method generalize to stochastic environments, particularly those with high entropy or noisy dynamics?

Key findings

• BeBold solves all 12 most challenging procedurally-generated MiniGrid tasks within 120 million environment steps, achieving a 100% success rate where prior SoTA solved only 50% of easier tasks.
• The method outperforms RND and RIDE on MonteZuma’s Revenge, achieving approximately 13,000 extrinsic return with an RNN-based model—surpassing RND’s 4,400.
• BeBold significantly mitigates the detachment problem observed in count-based methods, without requiring complex multi-stage training like Go-Explore.
• On NetHack, BeBold achieves state-of-the-art performance across multiple tasks, demonstrating strong generalization in highly complex, procedurally-generated environments.
• The intrinsic reward mechanism is asymptotically consistent, as inverse visitation counts vanish with sufficient exploration, avoiding the persistent rewards seen in state-difference methods.
• Despite its simplicity, BeBold outperforms more complex methods such as Go-Explore and RIDE, especially in environments requiring long-horizon exploration and boundary traversal.