[论文解读] Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

Adam Fourney, Gagan Bansal+18 more|arXiv (Cornell University)|Nov 7, 2024

Complex Systems and Decision Making被引用 11

一句话总结

Magentic-One 提供一个开源的通用型多智能体系统，由编排者（Orchestrator）驱动，规划、将任务分配给专门的智能体，并从错误中恢复，在 GAIA、AssistantBench 和 WebArena 基准测试上取得具竞争力的表现。它还引入 AutoGenBench，用于对智能体系统进行严格的代理评估。

ABSTRACT

Modern AI agents, driven by advances in large foundation models, promise to enhance our productivity and transform our lives by augmenting our knowledge and capabilities. To achieve this vision, AI agents must effectively plan, perform multi-step reasoning and actions, respond to novel observations, and recover from errors, to successfully complete complex tasks across a wide range of scenarios. In this work, we introduce Magentic-One, a high-performing open-source agentic system for solving such tasks. Magentic-One uses a multi-agent architecture where a lead agent, the Orchestrator, plans, tracks progress, and re-plans to recover from errors. Throughout task execution, the Orchestrator directs other specialized agents to perform tasks as needed, such as operating a web browser, navigating local files, or writing and executing Python code. We show that Magentic-One achieves statistically competitive performance to the state-of-the-art on three diverse and challenging agentic benchmarks: GAIA, AssistantBench, and WebArena. Magentic-One achieves these results without modification to core agent capabilities or to how they collaborate, demonstrating progress towards generalist agentic systems. Moreover, Magentic-One's modular design allows agents to be added or removed from the team without additional prompt tuning or training, easing development and making it extensible to future scenarios. We provide an open-source implementation of Magentic-One, and we include AutoGenBench, a standalone tool for agentic evaluation. AutoGenBench provides built-in controls for repetition and isolation to run agentic benchmarks in a rigorous and contained manner -- which is important when agents' actions have side-effects. Magentic-One, AutoGenBench and detailed empirical performance evaluations of Magentic-One, including ablations and error analysis are available at https://aka.ms/magentic-one

研究动机与目标

• 促使开发能够跨越多领域解决复杂任务的通用型代理系统.
• 提出一个模块化的多智能体架构，其中编排者（Orchestrator）协调专门的智能体（WebSurfer、FileSurfer、Coder、ComputerTerminal）。
• 为在有状态环境中对代理系统进行严格、可重复评估而引入 AutoGenBench。
• 证明一个未修改、模块化的多智能体设置即可在多个基准上达到与现有最先进水平的性能.
• 强调可扩展性、添加/移除智能体的便捷性，以及未来情景适应的潜力

提出的方法

• 提出 Magentic-One 的双循环编排：外循环维护任务账本（计划、事实、有根据的猜测）和内循环维护进度账本（任务分配与进度跟踪）。
• 利用编排者生成逐步计划并将子任务分配给专门的智能体，在遇到停滞或错误时进行重置/重新计划以恢复。
• 在共享控制流下部署一组专门的智能体（WebSurfer 负责浏览器/网页任务、FileSurfer 负责文件交互、Coder 负责代码/制品创建、ComputerTerminal 负责代码执行）。
• 通过基于计数的内循环和在卡住时的反思步骤引入恢复机制，从而实现有限的错误恢复。
• 将 AutoGenBench 作为一个独立的、受控的评估框架引入，以确保对代理系统的孤立、可重复基准测试。
• 在三个基准（GAIA、AssistantBench、WebArena）上进行评估，提供开源实现和消融分析。

Figure 1 : An illustration of the Magentic-One mutli-agent team completing a complex task from the GAIA benchmark. Magentic-One’s Orchestrator agent creates a plan, delegates tasks to other agents, and tracks progress towards the goal, dynamically revising the plan as needed. The Orchestrator can de

实验结果

研究问题

• RQ1 Magnetic-One 是否能够在各种代理任务基准（GAIA、AssistantBench、WebArena）上实现具有竞争性的任务完成率或准确性？
• RQ2 模块化多智能体设计（编排者加专门智能体）在不进行提示训练或核心架构变更的情况下，如何提升性能与泛化能力？
• RQ3 评估框架（AutoGenBench）对可复现性、安全性和代理任务方差的影响？
• RQ4 各个智能体对整体性能的独立贡献是什么，主要的失败模式在哪里？
• RQ5 在改变底层大型语言模型（如 GPT-4o 与 o1-preview）时，系统是否仍能维持性能？

主要发现

• Magentic-One 在 GAIA 任务完成率达到 38%、在 WebArena 为 32.8%、在 AssistantBench 为 27.7%，在 GPT-4o 与 o1 配置下的表现与最先进基线具有统计学意义的可比性。
• 在仅使用 GPT-4o 的情况下，Magentic-One 在所报告的设置中达到 GAIA 32.33%、AssistantBench EM 11.0%、AssistantBench 准确性 25.3%、WebArena 32.8%；结合 GPT-4o 与 o1，GAIA 提升至 38.00%、AssistantBench EM 13.3%、准确性 27.7。
• 消融分析显示各个智能体对整体性能具有附加价值，并揭示清晰的错误模式，为改进提供指引。
• AutoGenBench 通过确保固定初始条件、运行之间的隔离以及对有状态任务的安全评估来实现受控、可重复的基准测试。
• 结果证明一种通用型代理系统在网络和基于文件的任务上即可运行，而无需对智能体能力或协作方案进行核心变更。

Figure 2 : Magentic-One features an Orchestrator agent that implements two loops: an outer loop and an inner loop. The outer loop (lighter background with solid arrows) manages the task ledger (containing facts, guesses, and plan). The inner loop (darker background with dotted arrows) manages the pr