[Paper Review] CraftAssist: A Framework for Dialogue-enabled Interactive Agents

Jonathan Gray, Kavya Srinet+7 more|arXiv (Cornell University)|Jul 19, 2019

Multi-Agent Systems and Negotiation38 references18 citations

TL;DR

CraftAssist introduces a modular, dialogue-enabled framework for interactive agents in Minecraft, enabling human-bot collaboration through in-game chat and structured task execution. It provides open-source tools, annotated data (2,586 houses with semantic parses), and a pipelined architecture to study natural language understanding and learning from human interaction in open-ended environments.

ABSTRACT

This paper describes an implementation of a bot assistant in Minecraft, and the tools and platform allowing players to interact with the bot and to record those interactions. The purpose of building such an assistant is to facilitate the study of agents that can complete tasks specified by dialogue, and eventually, to learn from dialogue interactions.

Motivation & Objective

• To develop a flexible, interactive bot assistant in Minecraft that can understand and execute natural language commands.
• To enable research on agents that learn from dialogue interactions by providing a platform for collecting and labeling human-bot interaction data.
• To support the study of situated natural language understanding in open-ended, long-tailed task distributions through a modular, extensible framework.
• To facilitate collaborative research by releasing code, datasets, and infrastructure for data collection and bot deployment.

Proposed method

• The framework uses a client-server architecture to connect the bot to a Minecraft server via the Minecraft network protocol, enabling seamless multiplayer interaction.
• It implements a pipelined architecture with separate modules for natural language understanding, symbolic world state tracking, and action execution using a relational database.
• The bot interprets natural language commands via a semantic parser that maps utterances to logical forms, which are then executed using predefined action dictionaries.
• A modular design allows independent development and testing of components such as the parser, world state manager, and action executor.
• The system records human interactions, including commands, actions, and rephrasings, to generate training data for semantic parsing and dialogue learning.
• The platform includes tools for labeling house components (e.g., walls, roofs) and generating rephrased command templates for data augmentation.

Experimental results

Research questions

• RQ1 How can an interactive agent be built to understand and execute diverse, naturally-occurring language commands in a complex, open-ended environment like Minecraft?
• RQ2 What design patterns enable scalable and modular development of dialogue-enabled agents that can be extended with new capabilities?
• RQ3 How can human-bot interaction data be systematically collected and structured to support learning of semantic parsing and task execution?
• RQ4 What are the trade-offs between modular, pipeline-based architectures and end-to-end learning approaches in dialogue-enabled agent systems?
• RQ5 How can symbolic memory and structured world state representations improve interpretability and extensibility in interactive agents?

Key findings

• The framework successfully supports human-bot collaboration in Minecraft, with bots responding to natural language commands through in-game chat and performing complex construction tasks.
• The release of 2,586 human-built houses with detailed annotations enables large-scale training and evaluation of semantic parsing models.
• The modular architecture allows for independent development and extension of components, such as adding new actions or improving the semantic parser.
• The system enables data collection at scale, including rephrasings of templates and logical form mappings, supporting future end-to-end learning approaches.
• The symbolic memory model using a relational database improves debugging, integration of external knowledge (e.g., schematics), and reliable lookup operations.
• The platform has been open-sourced, with code, data, and infrastructure released to support community-driven research on interactive agents.