[Paper Review] Mastering Diverse Domains through World Models

Danijar Hafner, Jurgis Pašukonis+2 more|arXiv (Cornell University)|Jan 10, 2023

Machine Learning and Data ClassificationComputer Science91 citations

TL;DR

DreamerV3 is a general, scalable reinforcement learning algorithm that uses world models and fixed hyperparameters to master diverse domains, including Minecraft, across visual and non-visual inputs. Larger models improve data efficiency and final performance.

ABSTRACT

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a significant challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.

Motivation & Objective

• Aim to create a general reinforcement learning algorithm that masters diverse domains without task-specific tuning.
• Investigate whether fixed hyperparameters can enable robust learning across modalities, inputs, and reward structures.
• Examine scaling properties to understand how model size affects data-efficiency and performance.
• Demonstrate practical applicability by solving challenging tasks such as collecting diamonds in Minecraft from scratch.
• Provide extensive benchmarking and open training curves to facilitate comparison.

Proposed method

• Employ a world model (RSSM) that encodes observations into discrete z_t and predicts future representations, rewards, and continuations.
• Use three networks (world model, critic, actor) trained from replayed experience without gradient sharing.
• Apply symlog transformations to predictions, rewards, and critics to stabilize learning across domains with different signal magnitudes.
• Introduce fixed loss balancing with KL balancing and free bits, plus a scale-down of large returns to stabilize policy entropy with fixed hyperparameters.
• Adopt a discrete regression approach for the critic and reward predictor via twohot encoding of symlog-transformed targets for robust learning in sparse rewards.
• Normalize returns for the actor using a percentile-based scale (S) to maintain consistent exploration across dense and sparse rewards.

Figure 1 : Using the same hyperparameters across all domains, DreamerV3 outperforms specialized model-free and model-based algorithms in a wide range of benchmarks and data-efficiency regimes. Applied out of the box, DreamerV3 also learns to obtain diamonds in the popular video game Minecraft from s

Experimental results

Research questions

• RQ1 Can DreamerV3 master diverse domains with fixed hyperparameters across continuous, discrete, visual, and low-dimensional inputs?
• RQ2 How does increasing model size affect final performance and data efficiency in a fixed-hyperparameter regime?
• RQ3 Is it possible to collect Diamonds in Minecraft from scratch without human data or curricula using DreamerV3?
• RQ4 What are the scaling behaviors and practical guidelines for applying DreamerV3 to new tasks?

Key findings

• DreamerV3 outperforms specialized model-free and model-based algorithms across a wide range of domains using fixed hyperparameters.
• The algorithm demonstrates favorable scaling, with larger models yielding higher data efficiency and better final performance.
• DreamerV3 achieves state-of-the-art results on continuous control from states and images, on BSuite, and on Crafter.
• It is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula.
• Training curves and results are released to facilitate transparent comparisons across methods.
• Across seven benchmarks, DreamerV3 learns with fixed hyperparameters and demonstrates broad applicability.

(a) Control Suite