[Paper Review] RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments

Roberta Răileanu, Tim Rocktäschel|arXiv (Cornell University)|Feb 27, 2020

Reinforcement Learning in Robotics60 references38 citations

TL;DR

RIDE introduces an intrinsic reward that rewards actions causing impactful changes in a learned state representation, improving exploration in procedurally generated sparse-reward environments. It outperforms prior intrinsic rewards on MiniGrid tasks and generalizes across varied mazes.

ABSTRACT

Exploration in sparse reward environments remains one of the key challenges of model-free reinforcement learning. Instead of solely relying on extrinsic rewards provided by the environment, many state-of-the-art methods use intrinsic rewards to encourage exploration. However, we show that existing methods fall short in procedurally-generated environments where an agent is unlikely to visit a state more than once. We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state representation. We evaluate our method on multiple challenging procedurally-generated tasks in MiniGrid, as well as on tasks with high-dimensional observations used in prior work. Our experiments demonstrate that this approach is more sample efficient than existing exploration methods, particularly for procedurally-generated MiniGrid environments. Furthermore, we analyze the learned behavior as well as the intrinsic reward received by our agent. In contrast to previous approaches, our intrinsic reward does not diminish during the course of training and it rewards the agent substantially more for interacting with objects that it can control.

Motivation & Objective

• Motivate exploration in sparse-reward, procedurally generated environments where extrinsic rewards are rare.
• Develop an intrinsic reward that targets impactful state changes rather than mere novelty.
• Learn a state representation via forward and inverse dynamics to ground the reward in controllable aspects of the environment.
• Evaluate RIDE against standard and intrinsic exploration baselines across varied MiniGrid tasks and high-dimensional singleton environments.

Proposed method

• Learn a latent state representation phi(s) using forward and inverse dynamics models as in Pathak et al. (2017).
• Define an intrinsic reward R_IDE = ||phi(s_{t+1}) - phi(s_t)||_2 divided by sqrt(N_ep(s_{t+1})) to reward impactful transitions while discouraging trivial back-and-forth moves.
• Train forward and inverse models with losses L_fw and L_inv alongside RL objective L_RL, without letting RL updates influence the representation networks.
• Use both episodic state visitation counts to discount the intrinsic reward (epistemic grounding) and ensure the reward remains focused on controllable environment changes.
• Ground policy learning on top of the intrinsic reward while keeping the embedding networks separate from policy updates to avoid gaming the reward.

Experimental results

Research questions

• RQ1 Can an intrinsic reward based on impact in a learned latent space improve exploration in procedurally generated, sparse-reward RL tasks?
• RQ2 How does RIDE compare to count-based and curiosity-based intrinsic rewards in terms of sample efficiency and task solvability in MiniGrid and high-dimensional singleton tasks?
• RQ3 Do learned representations focus rewards on actions that interact with objects the agent can control, and is the reward signal persistent across training?
• RQ4 Does RIDE generalize better to procedurally generated environments than prior intrinsic motivation methods?

Key findings

• RIDE outperforms baseline exploration methods (Count, RND, ICM) and standard RL (IMPALA) on challenging MiniGrid tasks, solving harder environments where others fail.
• RIDE's intrinsic reward remains dynamic and does not diminish over 100M frames, unlike some curiosity-based or count-based bonuses.
• RIDE emphasizes actions that interact with controllable objects (e.g., opening doors) more than generic movement, as shown by intrinsic reward analyses.
• Training on procedurally generated mazes leads to broader exploration with RIDE than when using singleton mazes, indicating better generalization across environment instances.
• In singleton VizDoom and Mario benchmarks, RIDE performs comparably or better than baseline methods, while curiosity-based methods can hinder learning when combined with extrinsic rewards.