[Paper Review] Successor-Predecessor Intrinsic Exploration

Changmin Yu, Neil Burgess+2 more|arXiv (Cornell University)|May 24, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Successor-Predecessor Intrinsic Exploration (SPIE), a novel intrinsic exploration method that combines prospective successor representation (SR) and retrospective predecessor representation (PR) to guide efficient, structure-aware exploration in sparse-reward environments. SPIE improves sample efficiency and asymptotic performance over prior methods, achieving near-optimal performance on Montezuma’s Revenge with only 8×10⁶ frames—significantly faster than DQN-SR and RND.

ABSTRACT

Exploration is essential in reinforcement learning, particularly in environments where external rewards are sparse. Here we focus on exploration with intrinsic rewards, where the agent transiently augments the external rewards with self-generated intrinsic rewards. Although the study of intrinsic rewards has a long history, existing methods focus on composing the intrinsic reward based on measures of future prospects of states, ignoring the information contained in the retrospective structure of transition sequences. Here we argue that the agent can utilise retrospective information to generate explorative behaviour with structure-awareness, facilitating efficient exploration based on global instead of local information. We propose Successor-Predecessor Intrinsic Exploration (SPIE), an exploration algorithm based on a novel intrinsic reward combining prospective and retrospective information. We show that SPIE yields more efficient and ethologically plausible exploratory behaviour in environments with sparse rewards and bottleneck states than competing methods. We also implement SPIE in deep reinforcement learning agents, and show that the resulting agent achieves stronger empirical performance than existing methods on sparse-reward Atari games.

Motivation & Objective

• To address inefficient exploration in sparse-reward environments where local signals like visitation counts fail to guide agents toward structurally important states such as bottlenecks.
• To integrate retrospective information (predecessor structure) with prospective information (successor representation) to create a more globally aware intrinsic reward.
• To develop a general framework for intrinsic exploration that improves sample efficiency and asymptotic return in both discrete and continuous MDPs.
• To validate SPIE’s effectiveness in deep RL by integrating it with DQN and comparing performance against state-of-the-art methods like RND and DQN-SR.
• To explore the potential of SPIE for modeling biologically plausible, cycling-like exploratory behavior in structured environments.

Proposed method

• SPIE constructs an intrinsic reward by combining the successor representation (SR), which captures future state occupancy, and the predecessor representation (PR), which encodes expected past states.
• The intrinsic reward is defined as a weighted sum of SR and PR, enabling the agent to prioritize states that are both promising for future reachability and critical for connectivity in the state space.
• For discrete MDPs, SPIE uses tabular SR and PR to compute intrinsic rewards based on expected future and past state visits.
• For continuous MDPs, SPIE introduces 'predecessor features'—a neural network-based generalization of PR—to capture retrospective structure in high-dimensional state spaces.
• The method is integrated into deep Q-networks (DQN) by augmenting the external reward with the SPIE intrinsic reward, enabling online temporal difference learning.
• SPIE is designed to be modular and compatible with any deep RL agent, allowing plug-in use with existing algorithms.

Experimental results

Research questions

• RQ1 Can combining retrospective (predecessor) and prospective (successor) information improve exploration efficiency in sparse-reward environments?
• RQ2 Does SPIE enable agents to identify and traverse bottleneck states more effectively than methods relying only on local visitation counts or future prospects?
• RQ3 How does SPIE compare to state-of-the-art intrinsic exploration methods like RND and DQN-SR in terms of sample efficiency and asymptotic performance on Atari games?
• RQ4 Can SPIE produce ethologically plausible exploratory behavior, such as cycling between clusters in structured grid worlds, without external shaping?
• RQ5 What is the impact of integrating SPIE with deep RL agents on learning dynamics and final performance in continuous and discrete control tasks?

Key findings

• SPIE achieved near-asymptotic performance on Montezuma’s Revenge with only ~8×10⁶ training frames, whereas DQN-SR required ~2.4×10⁷ frames to reach a lower score.
• In the Freeway environment, SPIE outperformed DQN, DQN-MMC, RND, and DQN-SR, achieving a mean score of 27.5 (±0.2) compared to RND’s 28.2 and DQN-SR’s 29.4.
• On Gravitar, SPIE achieved a mean score of 1223.0 (±408.9), significantly outperforming RND (714.1) and DQN-SR (457.4).
• In Private Eye, SPIE reached 488.2 (±390.9), surpassing RND (61.3) and DQN-SR (104.4), demonstrating robustness in complex exploration tasks.
• SPIE exhibited ethologically plausible exploratory behavior in grid worlds, including cycling between distinct clusters of states, indicating structure-aware exploration.
• The integration of SPIE with DQN led to consistent improvements across all six hard-exploration Atari games tested, with superior sample efficiency and higher asymptotic returns than competing methods.