[Paper Review] Multi-Stage Episodic Control for Strategic Exploration in Text Games

Jens Tuyls, Shunyu Yao+2 more|arXiv (Cornell University)|Jan 4, 2022

Artificial Intelligence in Games4 citations

TL;DR

This paper proposes eXploit-Then-eXplore (XTX), a multi-stage episodic control algorithm that explicitly separates exploitation and exploration within each episode for strategic navigation in text-based games. By first exploiting promising past trajectories via self-imitation learning and then switching to curiosity-driven exploration using inverse dynamics, XTX achieves a 27% and 11% average improvement in normalized score over prior methods on the Jericho benchmark in deterministic and stochastic settings, respectively, including a 103 score on Zork1—over 2x higher than prior state-of-the-art.

ABSTRACT

Text adventure games present unique challenges to reinforcement learning methods due to their combinatorially large action spaces and sparse rewards. The interplay of these two factors is particularly demanding because large action spaces require extensive exploration, while sparse rewards provide limited feedback. This work proposes to tackle the explore-vs-exploit dilemma using a multi-stage approach that explicitly disentangles these two strategies within each episode. Our algorithm, called eXploit-Then-eXplore (XTX), begins each episode using an exploitation policy that imitates a set of promising trajectories from the past, and then switches over to an exploration policy aimed at discovering novel actions that lead to unseen state spaces. This policy decomposition allows us to combine global decisions about which parts of the game space to return to with curiosity-based local exploration in that space, motivated by how a human may approach these games. Our method significantly outperforms prior approaches by 27% and 11% average normalized score over 12 games from the Jericho benchmark (Hausknecht et al., 2020) in both deterministic and stochastic settings, respectively. On the game of Zork1, in particular, XTX obtains a score of 103, more than a 2x improvement over prior methods, and pushes past several known bottlenecks in the game that have plagued previous state-of-the-art methods.

Motivation & Objective

• To address the challenge of sparse rewards and combinatorially large action spaces in text adventure games, which hinder effective exploration and exploitation.
• To overcome the limitations of single-policy approaches that struggle to balance exploration and exploitation in dynamic, high-dimensional environments.
• To enable agents to strategically return to promising frontier states and perform targeted local exploration to overcome persistent game bottlenecks.
• To develop a method that scales across both deterministic and stochastic variants of text games without relying on environment-specific assumptions or external memory.
• To demonstrate that explicit policy decomposition within episodes leads to superior sample efficiency and performance compared to end-to-end or curiosity-augmented baselines.

Proposed method

• XTX uses a two-phase episodic control strategy: first exploiting via a policy trained with self-imitation learning on a mixture of high-scoring and long trajectories from past experience.
• The exploitation policy is a soft mixture of trajectories, preventing premature convergence to local optima and enabling robust return to frontier states.
• After exploitation, the agent switches to an exploration policy that combines a temporal difference (TD) loss with an auxiliary inverse dynamics loss to encourage discovery of novel, unseen actions.
• A single interpolation parameter λ controls the transition between exploitation and exploration, enabling smooth policy blending within a single episode.
• The exploration policy leverages value functions trained on both intrinsic curiosity and environment feedback, allowing strategic local exploration around known frontier states.
• The method does not require access to the underlying game simulator or game tree archives, making it scalable and generalizable across environments.

Experimental results

Research questions

• RQ1 Can a multi-stage episodic control approach that explicitly separates exploitation and exploration improve sample efficiency and performance in text-based games with sparse rewards and large action spaces?
• RQ2 How does the use of a mixture policy during exploitation prevent premature convergence to local optima in complex game environments?
• RQ3 To what extent does combining inverse dynamics-based curiosity with a dedicated exploitation policy enhance strategic exploration in unseen game states?
• RQ4 Does the proposed two-phase approach outperform single-policy methods with intrinsic curiosity bonuses or separate exploration policies in both deterministic and stochastic game settings?
• RQ5 How critical is the explicit separation of exploitation and exploration phases, as opposed to continuous mixing, for solving challenging text adventure games like Zork1?

Key findings

• XTX achieves a 27% average improvement in normalized score over prior methods on 12 deterministic games from the Jericho benchmark.
• In the stochastic setting, XTX improves average normalized score by 11% compared to prior baselines across the same 12 games.
• On Zork1, XTX attains a score of 103 in the deterministic setting—more than double the prior state-of-the-art score of 44.
• In the stochastic variant of Zork1, XTX reaches a score of 67, a significant improvement over the prior SOTA of 41.
• Ablation studies show that pure exploitation or pure exploration policies underperform, and that mixing policies in the exploitation phase is crucial for overcoming game-specific bottlenecks.
• The method successfully pushes past known bottlenecks in Zork1, such as defeating the troll in the dungeon, which previously blocked prior agents.