[Paper Review] CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant

Yacine Jernite, Kavya Srinet+2 more|arXiv (Cornell University)|Apr 17, 2019

Topic Modeling29 references8 citations

TL;DR

This paper introduces CraftAssist, a large-scale semantic parsing dataset of 35K human-generated natural language instructions with corresponding logical forms for controlling an agent in Minecraft. Using a structured grammar and crowd-sourced rephrasing, the authors train neural models that achieve high accuracy on template-generated data but face significant generalization challenges on free-form human instructions, highlighting the need for better robustness in instruction parsing systems.

ABSTRACT

We propose a large scale semantic parsing dataset focused on instruction-driven communication with an agent in Minecraft. We describe the data collection process which yields additional 35K human generated instructions with their semantic annotations. We report the performance of three baseline models and find that while a dataset of this size helps us train a usable instruction parser, it still poses interesting generalization challenges which we hope will help develop better and more robust models.

Motivation & Objective

• To develop a scalable, reproducible environment for training and evaluating semantic parsers in human-robot interaction using Minecraft as a simulation platform.
• To create a large-scale, annotated dataset of natural language instructions paired with executable logical forms for a Minecraft assistant.
• To evaluate neural semantic parsing models on both in-distribution and out-of-distribution instruction data, identifying generalization gaps in real-world usage.
• To enable end-to-end reproducibility by providing executable code for logical forms in-game, supporting full pipeline validation.

Proposed method

• Design a mid-level action grammar over Minecraft primitives (e.g., place, move, build) to define a structured logical form for agent actions.
• Generate synthetic natural language instructions from the grammar using templates, then collect crowd-sourced rephrasings to create diverse, human-like utterances.
• Collect a separate test set of free-form human-generated commands not based on templates, annotated with logical forms to evaluate generalization.
• Train and evaluate three neural semantic parsing models: a base Seq2Tree model, a SentenceRec model with recurrent context, and a reimplementation of Dong and Lapata (2016) adapted to the grammar.
• Use a 2-layer GRU encoder with FastText embeddings and learnable dimensions, applying label smoothing, dropout, and word dropout for regularization.
• Train models using Adagrad with early stopping on validation accuracy, selecting the best-performing model on rephrased validation data for final evaluation.

Experimental results

Research questions

• RQ1 Can neural semantic parsers generalize from template-generated training data to free-form human instructions in a complex, open-ended environment like Minecraft?
• RQ2 How do different model architectures (e.g., recurrent vs. independent prediction) perform under distribution shift from training to test data?
• RQ3 What types of errors do models make when parsing natural language instructions involving spatial references, object types, and action scopes?
• RQ4 To what extent can models correctly interpret ambiguous references such as 'the church' (schematic vs. reference object) or 'mobs' vs. 'objects'?
• RQ5 How does the inclusion of a dynamic, executable environment affect the evaluation and training of semantic parsing models?

Key findings

• All models achieved near-perfect accuracy (close to 100%) on template-generated training data, confirming they can invert the template generation process.
• The best model, SentenceRec, achieved 80.7% tree-level accuracy on rephrased validation data, indicating strong performance on paraphrased but in-distribution commands.
• Performance dropped significantly on prompts (86% valid under grammar), with models struggling most with domain shift from structured templates to imagined, open-ended commands.
• Models showed particular difficulty with span and categorical node prediction, especially when identifying the scope of location or repeat modifiers.
• Confusion between schematic objects (e.g., 'a house') and reference objects (e.g., 'the house') was common, indicating challenges in contextual understanding.
• The most frequent error type was predicting a node as inactive when it was actually present, particularly for span nodes, with depth/height confusion being a notable exception.