[Paper Review] JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

Zichuan Lin, Junyou Li+4 more|arXiv (Cornell University)|Dec 7, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

JueWu-MC proposes a sample-efficient hierarchical reinforcement learning framework for playing Minecraft, combining action-aware representation learning, discriminator-based self-imitation learning, and ensemble behavior cloning with consistency filtering. It achieved the highest performance score in the NeurIPS MineRL 2021 competition, outperforming all baselines with only 2.5 million training samples.

ABSTRACT

Learning rational behaviors in open-world games like Minecraft remains to be challenging for Reinforcement Learning (RL) research due to the compound challenge of partial observability, high-dimensional visual perception and delayed reward. To address this, we propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning to deal with perception and exploration. Specifically, our approach includes two levels of hierarchy, where the high-level controller learns a policy to control over options and the low-level workers learn to solve each sub-task. To boost the learning of sub-tasks, we propose a combination of techniques including 1) action-aware representation learning which captures underlying relations between action and representation, 2) discriminator-based self-imitation learning for efficient exploration, and 3) ensemble behavior cloning with consistency filtering for policy robustness. Extensive experiments show that JueWu-MC significantly improves sample efficiency and outperforms a set of baselines by a large margin. Notably, we won the championship of the NeurIPS MineRL 2021 research competition and achieved the highest performance score ever.

Motivation & Objective

• To address the compound challenges of partial observability, high-dimensional visual perception, and delayed rewards in open-world Minecraft environments.
• To improve sample efficiency in long-horizon decision-making by leveraging unstructured human demonstrations.
• To enhance exploration and policy robustness through novel representation and imitation learning techniques.
• To enable hierarchical decision-making by automatically extracting sub-goals from demonstrations and training low-level workers.
• To achieve state-of-the-art performance in the MineRL 2021 competition with minimal training samples.

Proposed method

• A two-level hierarchical reinforcement learning framework with a high-level controller managing options and low-level workers executing sub-tasks.
• Action-aware representation learning (A2RL) that captures dynamic relations between actions and visual representations using a learned mask network.
• Discriminator-based self-imitation learning (DSIL) that encourages exploration by reproducing past successful behaviors and focusing on high-return state distributions.
• Ensemble behavior cloning with consistency filtering to improve policy robustness by leveraging multiple expert demonstrations and filtering noisy trajectories.
• The high-level controller uses unstructured human demonstrations to automatically extract sub-goals and dispatch them to low-level workers.
• The framework integrates imitation learning and representation learning to boost sample efficiency and generalization in complex 3D environments.

Experimental results

Research questions

• RQ1 Can a hierarchical RL framework with automatic sub-goal discovery from unstructured demonstrations improve sample efficiency in long-horizon Minecraft tasks?
• RQ2 How does action-aware representation learning enhance visual understanding and control in 3D environments with high-dimensional observations?
• RQ3 To what extent does discriminator-based self-imitation learning improve exploration efficiency compared to standard self-imitation?
• RQ4 How does ensemble behavior cloning with consistency filtering enhance policy robustness in the presence of imperfect human demonstrations?
• RQ5 Can the integration of these techniques enable an agent to outperform existing baselines and win the MineRL 2021 competition?

Key findings

• JueWu-MC achieved the highest performance score in the NeurIPS MineRL 2021 competition, setting a new record with only 2.5 million training samples.
• The model reached a score of 100 using just 2.5 million samples, significantly fewer than the 8 million sample limit in the competition.
• Ablation studies showed that A2RL and EBC contributed more to performance than DSIL, with A2RL having earlier and broader impact on the learning pipeline.
• Saliency map visualization confirmed that the A2RL mask network effectively focuses on dynamic, action-relevant regions of the visual input, improving interpretability and control.
• DSIL enabled more compact and targeted exploration, with agents concentrating on high-return state distributions earlier in training compared to PPO and PPO+SIL.
• The framework outperformed all prior baselines in sample efficiency and task performance, demonstrating the effectiveness of combining hierarchical planning with advanced representation and imitation learning.