[论文解读] Mastering Diverse Domains through World Models

Danijar Hafner, Jurgis Pašukonis+2 more|arXiv (Cornell University)|Jan 10, 2023

Machine Learning and Data Classification被引用 91

一句话总结

DreamerV3 是一种通用、可扩展的强化学习算法，使用世界模型和固定超参数，在包括 Minecraft 在内的多种领域掌握视觉与非视觉输入的能力。更大模型提高数据效率和最终性能。

ABSTRACT

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a significant challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.

研究动机与目标

• 旨在创建一种通用的强化学习算法，在不进行任务特定调优的情况下掌握多样化领域。
• 研究固定超参数是否能够在模态、输入和奖励结构中实现鲁棒学习。
• 考察缩放属性，以理解模型大小如何影响数据效率和性能。
• 通过解决从零开始在 Minecraft 中收集钻石等具有挑战性的任务，展示其实用性。
• 提供广泛的基准测试和开放的训练曲线，以便进行比较。

提出的方法

• 采用世界模型（RSSM），将观测编码为离散的 z_t，并预测未来表征、奖励和继续状态。
• 使用三个网络（世界模型、评论家、行动者），从回放经验中训练，且不进行梯度共享。
• 对预测、奖励和评论家应用 symlog 变换，以在具有不同信号幅度的领域中稳定学习。
• 引入固定损失平衡，结合 KL 平衡和自由比特，并对较大的回报进行缩放，以在固定超参数下稳定策略熵。
• 通过对 symlog 转换目标进行 twohot 编码，为评论家和奖励预测器采用离散回归方法，在稀疏奖励下实现鲁棒学习。
• 使用基于百分位的尺度 (S) 对行动者的回报进行归一化，以在密集和稀疏奖励之间保持一致的探索。

Figure 1 : Using the same hyperparameters across all domains, DreamerV3 outperforms specialized model-free and model-based algorithms in a wide range of benchmarks and data-efficiency regimes. Applied out of the box, DreamerV3 also learns to obtain diamonds in the popular video game Minecraft from s

实验结果

研究问题

• RQ1 Can DreamerV3 master diverse domains with fixed hyperparameters across continuous, discrete, visual, and low-dimensional inputs?
• RQ2 How does increasing model size affect final performance and data efficiency in a fixed-hyperparameter regime?
• RQ3 Is it possible to collect Diamonds in Minecraft from scratch without human data or curricula using DreamerV3?
• RQ4 What are the scaling behaviors and practical guidelines for applying DreamerV3 to new tasks?

主要发现

• DreamerV3 在使用固定超参数的前提下，在广泛领域中超越了专门的基于模型和非基于模型的算法。
• 该算法显示出有利的缩放性，较大规模的模型带来更高的数据效率和更好的最终性能。
• DreamerV3 在状态和图像的连续控制、BSuite 和 Crafter 上取得了最先进的结果。
• 它是首个在 Minecraft 中从零开始收集 diamonds 而无需人类数据或课程设计的算法。
• 训练曲线和结果已发布，以促进跨方法的透明比较。
• 在七个基准测试中，DreamerV3 使用固定超参数进行学习，并展示了广泛的适用性。

(a) Control Suite