[论文解读] The MineRL BASALT Competition on Learning from Human Feedback

Rohin Shah, Cody Wild+11 more|arXiv (Cornell University)|Jul 5, 2021

Reinforcement Learning in Robotics参考文献 31被引用 4

一句话总结

MineRL BASALT 竞赛提出了一项基准，用于通过人类反馈而非预定义奖励函数来训练人工智能智能体，重点在于 Minecraft 中复杂且以自然语言描述的任务。该基准通过人类判断评估智能体的任务完成情况，推动了价值对齐以及在开放环境中的可扩展模仿学习与偏好学习研究。

ABSTRACT

The last decade has seen a significant increase of interest in deep learning research, with many public successes that have demonstrated its potential. As such, these systems are now being incorporated into commercial products. With this comes an additional challenge: how can we build AI systems that solve tasks where there is not a crisp, well-defined specification? While multiple solutions have been proposed, in this competition we focus on one in particular: learning from human feedback. Rather than training AI systems using a predefined reward function or using a labeled dataset with a predefined set of categories, we instead train the AI system using a learning signal derived from some form of human feedback, which can evolve over time as the understanding of the task changes, or as the capabilities of the AI system improve. The MineRL BASALT competition aims to spur forward research on this important class of techniques. We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions. These tasks are defined by a paragraph of natural language: for example, "create a waterfall and take a scenic picture of it", with additional clarifying details. Participants must train a separate agent for each task, using any method they want. Agents are then evaluated by humans who have read the task description. To help participants get started, we provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline that leverages these demonstrations. Our hope is that this competition will improve our ability to build AI systems that do what their designers intend them to do, even when the intent cannot be easily formalized. Besides allowing AI to solve more tasks, this can also enable more effective regulation of AI systems, as well as making progress on the value alignment problem.

研究动机与目标

• 推进从人类反馈中学习（LfHF）的研究，作为手工设计奖励函数的替代方案。
• 解决在任务规范模糊或定义不清的情况下训练人工智能智能体的挑战。
• 创建一项基准，基于人类对任务完成情况的判断来评估智能体，而非仅基于奖励最大化。
• 推动可扩展、与人类对齐的人工智能力，能够理解并执行复杂且自然语言描述的指令。
• 通过整合多样化反馈模态来减少奖励欺骗和对齐偏差，从而体现人类意图。

提出的方法

• 参赛者可使用任意方法训练智能体，且任务不提供预定义奖励函数。
• 任务通过自然语言描述定义，例如“建造一个瀑布并拍摄一张风景照”。
• 人类评估者根据任务描述判断智能体表现，确保行为与预期目标一致。
• 提供人类示范数据集，以帮助参赛者初始化策略，支持模仿学习基线方法。
• 竞赛使用 MineRL 环境，这是一个丰富且开放的环境，支持多样化的智能体行为与目标。
• 评估通过人类评分员进行，他们评估智能体完成任务的完成度，从而降低环境设计带来的偏差。

实验结果

研究问题

• RQ1 通过人类反馈训练的智能体在开放环境中对复杂、自然语言描述的任务的泛化能力如何？
• RQ2 在奖励函数模糊或具有误导性的情况下，从人类反馈中学习是否优于传统的奖励塑造方法？
• RQ3 在复杂且类似现实世界任务中，模仿学习与基于偏好的方法在多大程度上提升了与人类意图的一致性？
• RQ4 人类对智能体行为的评估与自动化奖励信号在衡量任务成功方面相比如何？
• RQ5 在开放世界环境中，哪些反馈模态（例如示范、对比）能带来最稳健且对齐的智能体行为？

主要发现

• 该竞赛成功证明，人类反馈可用于在无预定义奖励函数的情况下训练智能体完成 Minecraft 中的复杂自然语言任务。
• 基于所提供人类示范的模仿学习基线方法训练出的智能体表现中等，表明其为后续改进提供了良好起点。
• 人类评估揭示，智能体常未能理解任务描述中的细微之处，凸显了改进对齐技术的必要性。
• Minecraft 的开放性使其成为评估 LfHF 方法的理想测试平台，因为当使用奖励函数时，智能体经常因意外行为而被奖励。
• 通过专用的 Discord 论坛，竞赛促进了协作研究社区的形成，增强了参赛者的参与度与知识共享。
• 将人类评估作为主要度量指标，降低了环境设计带来的偏差，并更真实地反映了现实世界中的任务成功。