[Paper Review] The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors

William H. Guss, Codel, Cayden+10 more|arXiv (Cornell University)|Apr 22, 2019

Reinforcement Learning in RoboticsComputer Science35 references47 citations

TL;DR

The paper proposes the MineRL competition and dataset to advance sample-efficient reinforcement learning by leveraging human demonstrations in a Minecraft environment, with a primary ObtainDiamond task and held-out evaluation to assess generalization under strict resource constraints.

ABSTRACT

Though deep reinforcement learning has led to breakthroughs in many difficult domains, these successes have required an ever-increasing number of samples. As state-of-the-art reinforcement learning (RL) systems require an exponentially increasing number of samples, their development is restricted to a continually shrinking segment of the AI community. Likewise, many of these systems cannot be applied to real-world problems, where environment samples are expensive. Resolution of these limitations requires new, sample-efficient methods. To facilitate research in this direction, we introduce the MineRL Competition on Sample Efficient Reinforcement Learning using Human Priors. The primary goal of the competition is to foster the development of algorithms which can efficiently leverage human demonstrations to drastically reduce the number of samples needed to solve complex, hierarchical, and sparse environments. To that end, we introduce: (1) the Minecraft ObtainDiamond task, a sequential decision making environment requiring long-term planning, hierarchical control, and efficient exploration methods; and (2) the MineRL-v0 dataset, a large-scale collection of over 60 million state-action pairs of human demonstrations that can be resimulated into embodied trajectories with arbitrary modifications to game state and visuals. Participants will compete to develop systems which solve the ObtainDiamond task with a limited number of samples from the environment simulator, Malmo. The competition is structured into two rounds in which competitors are provided several paired versions of the dataset and environment with different game textures. At the end of each round, competitors will submit containerized versions of their learning algorithms and they will then be trained/evaluated from scratch on a hold-out dataset-environment pair for a total of 4-days on a prespecified hardware platform.

Motivation & Objective

• Motivate development of sample-efficient RL methods that leverage human demonstrations to reduce environmental samples.
• Introduce the Minecraft ObtainDiamond task as a challenging, hierarchically structured environment.
• Release the MineRL-v0 dataset with over 60 million human demonstrations to enable imitation learning in embodied agents.
• Provide a two-round competition structure with hold-out evaluation to ensure fair benchmarking under fixed compute budgets.

Proposed method

• Define the primary ObtainDiamond task requiring long-horizon planning and exploration in Minecraft.
• Provide a large-scale MineRL-v0 dataset of state-action trajectories with rich annotations and hierarchical labels.
• Render demonstrations with varying textures and lighting to enable robust evaluation across environments.
• Offer baseline implementations and open-source tooling (Gym interface, data loader, Docker) to participants.
• Use AICrowd orchestration and fixed compute environments to enforce sample-efficiency evaluation.

Experimental results

Research questions

• RQ1 Can imitation learning and human priors dramatically reduce the number of environment samples needed to solve a complex, sparse-reward task?
• RQ2 How well do reinforcement learning methods leverage large-scale human demonstration datasets in a hierarchical, embodied domain like Minecraft?
• RQ3 What are the effects of different environment textures and visuals on training efficiency and policy performance under fixed compute budgets?
• RQ4 How do baseline RL methods compare to human performance under strict sample and compute constraints in ObtainDiamond?

Key findings

• Preliminary results show that methods leveraging human data improve sample efficiency across environments.
• Humans outperform all tested RL methods on the presented tasks, highlighting the challenge of long-horizon credit assignment in ObtainDiamond and related tasks.
• Treechop, Navigate (Sparse), and other environments reveal large gaps between RL baselines and human performance.
• Expert demonstrations enable higher rewards with fewer samples, especially in exploration-challenging settings like Navigate (Sparse).
• Imitation-based baselines (Behavioral Cloning, PreDQN) and pretrained variants demonstrate potential gains over non-pretrained RL methods.