[Paper Review] MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Linxi Fan, Guanzhi Wang+8 more|arXiv (Cornell University)|Jun 17, 2022

Topic Modeling59 citations

TL;DR

The paper introduces MineDojo, a Minecraft-based framework combining open-ended tasks, a large-scale multimodal knowledge base, and a learned reward function (MineCLIP) to train generally capable embodied agents without hand-crafted dense rewards.

ABSTRACT

Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite, knowledge bases, algorithm implementation, and pretrained models (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.

Motivation & Objective

• Propose a framework for open-ended, generally capable embodied agents built on three pillars: diverse environments, internet-scale knowledge, and scalable architectures.
• Introduce MineDojo’s Minecraft-based simulation with thousands of tasks and an automated, crowd-sourced knowledge base.
• Develop a learning algorithm that leverages large pre-trained video-language models as learned rewards to train agents without manual dense shaping rewards.
• Show that MineCLIP-based rewards enable competitive performance to manually engineered rewards on programmatic tasks and robust zero-shot generalization on unseen visuals.

Proposed method

• Provide a unified Minecraft-based simulator with standardized observation/action spaces across thousands of tasks.
• Construct two task suites: Programmatic tasks with ground-truth success criteria and Creative tasks evaluated with a learned metric.
• Build an internet-scale knowledge base from 730K+ YouTube Minecraft videos, 6K+ Wiki pages, and 340K+ Reddit posts to ground learning.
• Pre-train MineCLIP, a video-language contrastive model, on 16-frame video clips and transcripts to act as a dense, language-conditioned reward function.
• Integrate MineCLIP rewards into a PPO-based RL pipeline with Self-Imitation Learning to improve sample efficiency.
• Evaluate zero-shot and few-shot generalization and compare against manually engineered rewards and CLIP baselines.

Experimental results

Research questions

• RQ1 Can an embodied agent be trained to perform a wide range of open-ended Minecraft tasks using a learned, language-conditioned reward from internet-scale video data?
• RQ2 Does MineCLIP provide competitive or superior guidance compared to manually engineered rewards on programmatic tasks?
• RQ3 How well does an agent trained with MineCLIP rewards generalize to unseen terrains, weathers, and tasks?
• RQ4 What is the impact of multi-task training (12 tasks) on individual task performance and transfer effects?
• RQ5 Can the knowledge base and task suite scale collaboration and open-ended learning for embodied agents?

Key findings

• MineCLIP rewards achieve competitive performance to manual dense rewards on programmatic tasks and outperform CLIP OpenAI baselines.
• On 12-task multi-group experiments, MineCLIP-guided agents reach high success on several tasks and show robustness to visual distribution shifts.
• For Creative tasks, MineCLIP provides an automatic evaluation metric with high agreement to human judgments (F1 scores near human accuracy in the reported subset).
• A single agent trained on all 12 tasks shows mixed transfers, with some tasks improving and others declining compared to task-specific agents.
• Zero-shot generalization tests indicate MineCLIP-augmented agents maintain superior robustness to unseen terrains and conditions versus baselines.