[Paper Review] TextWorld: A Learning Environment for Text-based Games

Marc-Alexandre Côté, Ákos Kádár+11 more|arXiv (Cornell University)|Jun 29, 2018

Reinforcement Learning in Robotics26 references96 citations

TL;DR

TextWorld is a Python framework for training and evaluating RL agents on text-based games, enabling handcrafted or automatically generated games with controllable difficulty, language, and observability.

ABSTRACT

We introduce TextWorld, a sandbox learning environment for the training and evaluation of RL agents on text-based games. TextWorld is a Python library that handles interactive play-through of text games, as well as backend functions like state tracking and reward assignment. It comes with a curated list of games whose features and challenges we have analyzed. More significantly, it enables users to handcraft or automatically generate new games. Its generative mechanisms give precise control over the difficulty, scope, and language of constructed games, and can be used to relax challenges inherent to commercial text games like partial observability and sparse rewards. By generating sets of varied but similar games, TextWorld can also be used to study generalization and transfer learning. We cast text-based games in the Reinforcement Learning formalism, use our framework to develop a set of benchmark games, and evaluate several baseline agents on this set and the curated list.

Motivation & Objective

• Survey machine-learning challenges in text-based games and analyze hand-authored games.
• Describe the TextWorld framework, its features, and usage for RL research.
• Provide an initial set of text-based benchmark games for RL evaluation.
• Evaluate baseline RL agents on both benchmark and hand-authored games.
• Promote TextWorld as a living resource for curriculum, generalization, and transfer learning studies.

Proposed method

• Frame text-based games within the RL formalism (MDP/POMDP).
• Introduce a two-component architecture: a game generator and a game engine that handles interactive play.
• Use a logic-based inference engine to build valid game states and transitions via forward/backward chaining.
• Generate games via maps created with Random Walk and quests generated through dependency constraints and backward/forward mechanisms.
• Render game states to natural language descriptions using a CFG-driven text generator.
• Provide interfaces to compile generated games to Inform 7/Glulx and to interact with agents through a unified API.
• Offer optional intermediate rewards and tunable observability to study curriculum, generalization, and transfer.

Experimental results

Research questions

• RQ1 How can text-based games be formalized for RL research as controllable benchmarks?
• RQ2 How does TextWorld enable generation of varied but related games to study generalization and transfer learning?
• RQ3 What baseline RL methods perform on curated and generated text-based benchmarks, and what are their limitations?
• RQ4 How do controllable aspects like quest length, map size, and vocabulary affect learning dynamics and language understanding?
• RQ5 What are effective strategies (e.g., intermediate rewards) to facilitate training in sparse-reward, partially observable text environments?

Key findings

• TextWorld provides a scalable framework to generate and interact with text-based games for RL research.
• The framework enables exact state-tracking and intermediate rewards through a structured representation of game states and a winning policy.
• TextWorld supports curriculum and transfer learning experiments by varying environment parameters and generating related games.
• A curated list of hand-authored text games and an initial benchmark set are introduced, with baseline agents evaluated on them.
• The authors position TextWorld as a living resource intended to evolve with community contributions and new benchmarks.