[Paper Review] The MineRL BASALT Competition on Learning from Human Feedback

Rohin Shah, Cody Wild+11 more|arXiv (Cornell University)|Jul 5, 2021

Reinforcement Learning in Robotics31 references4 citations

TL;DR

The MineRL BASALT competition proposes a benchmark for training AI agents using human feedback instead of predefined reward functions, focusing on complex, natural language-described tasks in Minecraft. It evaluates agents via human judgment on task completion, advancing research in value alignment and scalable imitation and preference learning in open-ended environments.

ABSTRACT

The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are now being incorporated into commercial products. With this comes an additional challenge: how can we build AI systems that solve tasks where there is not a crisp, well-defined specification? While multiple solutions have been proposed, in this competition we focus on one in particular: learning from human feedback. Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve. The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. These tasks are defined by a paragraph of natural language: for example, "create a waterfall and take a scenic picture of it", with additional clarifying details. Participants must train a separate agent for each task, using any method they want. Agents are then evaluated by humans who have read the task description. To help participants get started, we provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline that leverages these demonstrations. Our hope is that this competition will improve our ability to build AI systems that do what their designers intend them to do, even when the intent cannot be easily formalized. Besides allowing AI to solve more tasks, this can also enable more effective regulation of AI systems, as well as making progress on the value alignment problem.

Motivation & Objective

• To advance research in learning from human feedback (LfHF) as an alternative to handcrafted reward functions.
• To address the challenge of training AI agents on tasks with ambiguous or ill-defined specifications.
• To create a benchmark that evaluates agents based on human judgment of task completion, not just reward maximization.
• To promote scalable, human-aligned AI systems capable of understanding and executing complex, natural language instructions.
• To reduce reward hacking and misalignment by incorporating human intent through diverse feedback modalities.

Proposed method

• Participants train agents using any method they prefer, with no predefined reward function provided for the tasks.
• Tasks are defined by natural language descriptions such as 'create a waterfall and take a scenic picture of it'.
• Human evaluators judge agent performance based on the task description, ensuring alignment with intended behavior.
• A dataset of human demonstrations is provided to help participants initialize policies, supporting imitation learning baselines.
• The competition uses the MineRL environment, a rich, open-ended world that supports diverse agent behaviors and goals.
• Evaluation is conducted via human raters who assess how well agents complete tasks, reducing bias from environment design.

Experimental results

Research questions

• RQ1 How well can agents trained via human feedback generalize to complex, natural language-described tasks in open-ended environments?
• RQ2 Can learning from human feedback outperform traditional reward shaping in scenarios where reward functions are ambiguous or misleading?
• RQ3 To what extent do imitation learning and preference-based methods improve alignment with human intent in complex, real-world-like tasks?
• RQ4 How does human evaluation of agent behavior compare to automated reward signals in measuring task success?
• RQ5 What feedback modalities (e.g., demonstrations, comparisons) lead to the most robust and aligned agent behavior in open-world settings?

Key findings

• The competition successfully demonstrated that human feedback can be used to train agents to complete complex, natural language tasks in Minecraft without predefined reward functions.
• Agents trained using imitation learning baselines from provided human demonstrations achieved moderate performance, indicating a strong starting point for further improvement.
• Human evaluation revealed that agents often failed to understand subtle aspects of task descriptions, highlighting the need for better alignment techniques.
• The open-ended nature of Minecraft made it a strong testbed for evaluating LfHF methods, as agents were frequently rewarded for unintended behaviors when reward functions were used.
• The competition fostered a collaborative research community through a dedicated Discord forum, enhancing participant engagement and knowledge sharing.
• The use of human evaluation as the primary metric reduced bias from environment design and better reflected real-world task success.