[Paper Review] Hierarchical Decision Making by Generating and Following Natural Language Instructions

Hengyuan Hu, Denis Yarats+3 more|arXiv (Cornell University)|Jan 1, 2019

Topic Modeling20 citations

TL;DR

This paper proposes a hierarchical decision-making framework that uses natural language as a latent plan to coordinate complex, long-horizon actions in a real-time strategy game. By generating and following human-like instructions, the model outperforms direct behavior cloning, demonstrating that language's compositional structure enables more effective and generalizable control of multi-agent systems.

ABSTRACT

We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making. Rather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Experiments show that models using natural language as a latent variable significantly outperform models that directly imitate human actions. The compositional structure of language proves crucial to its effectiveness for action representation. We also release our code, models and data.

Motivation & Objective

• To enable hierarchical decision making in long-horizon, multi-agent environments by representing complex actions as natural language instructions.
• To investigate whether natural language can serve as a more expressive and compositional representation than direct action sequences for policy learning.
• To develop and evaluate a two-stage framework where an instructor generates instructions and an executor performs them, improving generalization and sample efficiency.
• To collect and release a large-scale dataset of 76k human-generated instruction-execution pairs for training and evaluation.

Proposed method

• The framework uses a two-stage architecture: an instructor model generates natural language instructions, and a separate executor model interprets and executes them.
• The instructor model is trained to produce high-level, compositional instructions that describe multi-step goals.
• The executor model is trained to map instructions to executable actions, learning to interpret linguistic structure.
• The approach is evaluated in a custom real-time strategy game environment requiring coordination of many units over extended time scales.
• A dataset of 76,000 human-played instruction-execution pairs is collected to supervise both instructor and executor training.
• The method leverages the compositional nature of language to generalize across unseen task structures and improve zero-shot transfer.

Experimental results

Research questions

• RQ1 Can natural language serve as an effective and compositional latent representation for hierarchical decision making in complex, long-horizon environments?
• RQ2 Does generating and following natural language instructions lead to better performance than direct imitation of human actions in multi-agent coordination tasks?
• RQ3 How does the compositional structure of language contribute to generalization and sample efficiency in policy learning?
• RQ4 To what extent can a separate executor model generalize to new instructions not seen during training?

Key findings

• Models using natural language as a latent variable significantly outperform direct behavior cloning baselines in multi-agent coordination tasks.
• The compositional structure of language enables better generalization to unseen task compositions and longer-horizon planning.
• The instructor-executor framework improves sample efficiency and enables zero-shot transfer to new instruction types.
• The human-annotated dataset of 76,000 instruction-execution pairs enables effective training and evaluation of the two-stage system.
• The executor model generalizes to new instructions by leveraging semantic and syntactic structure, demonstrating robustness to distributional shift.
• The approach achieves superior performance in long-horizon real-time strategy gameplay compared to end-to-end action imitation.