[论文解读] JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models

Zihao Wang, Shaofei Cai+10 more|arXiv (Cornell University)|Nov 10, 2023

Multimodal Machine Learning Applications被引用 6

一句话总结

JARVIS-1 是一种基于记忆增强的多模态语言模型的智能体，通过视觉和文本输入，在开放世界《我的世界》环境中实现了稳健的长时程规划与控制。通过整合存储过去成功计划与经验的多模态记忆，其在长时程任务 'ObtainDiamondPickaxe' 上取得了 12.5% 的成功率，是先前最先进智能体的五倍，展现出无需额外训练即可实现自我改进的终身学习能力。

ABSTRACT

Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. JARVIS-1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., "chopping trees" to long-horizon tasks, e.g., "obtaining a diamond pickaxe". JARVIS-1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task of $ exttt{ObtainDiamondPickaxe}$, JARVIS-1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. The project page is available at https://craftjarvis.org/JARVIS-1

研究动机与目标

• 开发一种通用智能体，能够在无需任务特定微调的情况下，在《我的世界》等开放世界环境中执行多样化、长时程的任务。
• 解决现有基于大语言模型的智能体在多模态感知、长期规划和持续学习方面的局限性。
• 通过自主任务提议和基于经验的记忆检索，实现终身自我改进的行为。
• 通过多模态记忆系统结合预训练知识与实时交互经验，提升规划的鲁棒性。

提出的方法

• JARVIS-1 基于混合架构构建，结合 MineCLIP 实现视觉-语义理解，以及大语言模型（LLM）实现高层规划。
• 它通过融合的多模态语言模型（MLM）处理多模态输入——图像和自然语言指令，生成特定任务的行动规划。
• 引入多模态记忆系统，用于存储和检索过往成功的计划与环境上下文，实现基于预训练知识和真实游戏经验的上下文学习。
• 该智能体使用目标条件控制器执行计划，将高层规划转化为《我的世界》环境中的低层动作。
• JARVIS-1 采用自我指导机制，在游戏过程中自主提出新任务，促进持续探索与经验积累。
• 该系统无需演示，通过经验回放和记忆增强推理，随时间推移持续提升性能。

Figure 1: How does JARVIS -1 unlock the technology tree of the Minecraft universe. JARVIS -1 can consistently obtain high-level items on the main tech-tree of the overworld in Minecraft, such as diamond, redstone, and golden items, which require collecting over 10 different intermediate items. JARVI

实验结果

研究问题

• RQ1 基于多模态语言模型的智能体是否能在开放世界《我的世界》环境中，对 200 多项多样化任务实现稳健且可泛化的规划？
• RQ2 与标准的大语言模型智能体相比，集成多模态记忆在长时程规划性能上有多大提升？
• RQ3 在终身学习设置中，智能体通过自主任务提议和经验积累，其自我改进能力在多大程度上得以实现？
• RQ4 即使没有额外训练，智能体的性能是否会随着游戏时长的增加而提升？

主要发现

• JARVIS-1 在长时程任务 'ObtainDiamondPickaxe' 上取得了 12.5% 的成功率，是先前最先进智能体 VPT（2.5%）的五倍。
• 该智能体在短时程任务上表现出近乎完美的性能，表明其具备强大的泛化能力和执行准确性。
• 长时程任务的性能随着游戏时间的增加而持续提升，即使未进行额外训练，表明其具备有效的经验积累与学习能力。
• 多模态记忆显著提升了规划的一致性与正确性，尤其在复杂、多步骤任务中表现突出。
• 在终身学习实验中，JARVIS-1 成功自主提出了新任务，收集了经验，并通过自主探索提升了其规划能力。
• 该智能体能够制作几乎所有主世界中的钻石级物品，包括钻石盔甲，证实其掌握了完整的科技树。

Figure 2: Challenges in open-world environments and how does JARVIS -1 tackle them . (Left) With situation-aware planning, JARVIS -1 substantially improves the success rate on the challenging ObtainDiamond task, compared to the baseline (GPT) without it. Note: Due to resource constraints, we can onl