[论文解读] Open-Ended Learning Leads to Generally Capable Agents

Open Ended Learning Team, Adam Stooke+16 more|arXiv (Cornell University)|Jul 27, 2021

Reinforcement Learning in Robotics参考文献 74被引用 55

一句话总结

本论文提出 XLand，是一个大规模多任务的三维环境和一个开放式训练循环，能够产生具零-shot泛化和跨多样任务空间的广泛能力的智能体，优于固定分布的强化学习。它展示了动态任务生成和迭代生成与策略蒸馏推动持续学习和涌现行为。

ABSTRACT

In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger scale transfer of behaviour through cheap finetuning.

研究动机与目标

• 促使在一个巨大且程序化生成的环境中，超越单一任务的智能体泛化。
• 定义并研究一个融合世界、游戏与共同玩家策略的环境空间（XLand），以创建一个大规模、平滑变化的任务空间。
• 开发一个开放式训练过程，持续改变任务分布与目标，以在性能分位点上维持学习。
• 用标准化分数分位数衡量进展，并分析在保留评估任务上的涌现一般行为。

提出的方法

• 介绍 XLand：一个原生多智能体、程序生成的三维世界空间，可控的智能体、对象、装置与奖励。
• 将任务表示为世界、游戏和共玩家策略，形成一个广阔且在多样性方面丰富的任务空间。
• 通过带有注意力网络的深度强化学习进行训练，该网络隐式建模游戏目标，使用动态生成的训练任务。
• 使用迭代的、基于种群的训练制度，通过跨代的策略蒸馏来启动新策略并重新定义性能前沿。
• 以评估空间的标准化分数分位来衡量进展，并将动态任务生成与均匀采样进行比较。

实验结果

研究问题

• RQ1 在一个开放式、动态生成的任务空间中训练的智能体是否能够在保留评估任务上实现零-shot泛化？
• RQ2 在使学习跨越广阔任务空间的连续性方面，动态变化的训练任务是否优于固定分布？
• RQ3 在以开放式学习训练具有广泛能力的智能体时，会出现哪些涌现的启发式和多智能体行为？
• RQ4 在这一开放式框架中，微调在零-shot 学习后能在多大程度上提升性能？

主要发现

• 智能体在包括 Hide and Seek、Capture the Flag、Tag 在内的多样化评估水平上展示零-shot 泛化。
• 对新任务进行约 1 亿步微调相较于零-shot 或从头训练可带来显著的性能提升。
• 在评估场景中出现了有向探索、通过其他玩家进行信息获取以及合作动态等涌现行为。
• 动态任务生成对学习至关重要，优于来自任务空间的均匀采样。
• 智能体的一般能力表明通过成本低廉的微调实现行为迁移的潜力更大。