[Paper Review] IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents

Artem Zholus, Alexey Skrynnik+6 more|arXiv (Cornell University)|May 31, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

IGLU Gridworld is a fast, lightweight, Python-based reinforcement learning environment for training and evaluating language-conditioned embodied agents in a 3D blocks-building task with collaborative dialog. It enables research in hierarchical RL, skill learning, and generalization through a multitask hierarchical baseline that outperforms random agents and prior competition solutions on diverse task types.

ABSTRACT

We present the IGLU Gridworld: a reinforcement learning environment for building and evaluating language conditioned embodied agents in a scalable way. The environment features visual agent embodiment, interactive learning through collaboration, language conditioned RL, and combinatorically hard task (3d blocks building) space.

Motivation & Objective

• To develop a scalable, fast, and transparent environment for training and evaluating embodied agents that follow natural language instructions.
• To support research in collaborative learning, generalization in imitation and reinforcement learning, and hierarchical decision-making in embodied AI.
• To enable open-ended, lifelong learning through a formalizable 3D grid-based task space with combinatorially rich structures.
• To provide a benchmark environment that supports skill-based task decomposition and curriculum learning.
• To facilitate data collection and model evaluation via a uniform, extensible Python API.

Proposed method

• The environment is implemented in pure Python with a decoupled renderer, enabling headless operation and high-speed simulation (up to 17,000 SPS when rendering is disabled).
• Observations include a 64×64×3 POV image, agent inventory, 3D building zone grid (11×9×11), and agent pose (position, pitch, yaw).
• The action space consists of 14 discrete actions: movement, jump, block placement/breaking, block type selection (1–6), and continuous camera control.
• Reward is computed via a convolution-like alignment procedure between the current state grid and the target grid, maximizing per-block intersection over time.
• Tasks are structured as sequences of collaboration segments, each defined by a (dialog context, target instruction, and target block configuration) pair.
• A multitask hierarchical RL baseline (MHB) is proposed, comprising a BERT-based NLP module for target prediction, a heuristic subtask generator, and an APPO-based RL module for subtask execution.

Experimental results

Research questions

• RQ1 How can a lightweight, fast, and scalable environment support the training of language-conditioned embodied agents in complex 3D tasks?
• RQ2 To what extent can hierarchical RL with curriculum learning improve generalization across diverse 3D block-building tasks?
• RQ3 Can a modular MHB architecture with NLP, heuristic planning, and RL components outperform random and baseline agents in multi-skill environments?
• RQ4 How does the environment support skill-based generalization and lifelong learning in embodied AI?
• RQ5 What is the performance of a multitask hierarchical agent across different task types, including flying, tall, diagonal, flat, and tricky structures?

Key findings

• IGLU Gridworld achieves up to 17,000 steps per second in headless mode, making it one of the fastest Python-based RL environments for 3D embodied tasks.
• The multitask hierarchical baseline (MHB) achieves an F1 score of 0.313 across all tasks, significantly outperforming the random agent (0.039) and showing strong performance on complex structures.
• The MHB agent with NLP head achieves an F1 score of 0.313 on all tasks, indicating strong generalization across diverse skill types.
• The MHB agent with full pipeline (including heuristic subtask generator) achieves an F1 score of 0.258 across all tasks, demonstrating the value of structured subtask planning.
• The environment supports rich task generalization through combinatorially hard 3D block configurations and interactive dialog-based collaboration.
• The MHB baseline outperforms all solutions submitted to the IGLU 2021 NeurIPS competition, validating its effectiveness.