[Paper Review] Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

Adam Fourney, Gagan Bansal+18 more|arXiv (Cornell University)|Nov 7, 2024

Complex Systems and Decision Making11 citations

TL;DR

Magentic-One presents an open-source generalist multi-agent system steered by an Orchestrator to plan, delegate tasks to specialized agents, and recover from errors, achieving competitive performance on GAIA, AssistantBench, and WebArena benchmarks. It also introduces AutoGenBench for rigorous agentic evaluation.

ABSTRACT

Modern AI agents, driven by advances in large foundation models, promise to enhance our productivity and transform our lives by augmenting our knowledge and capabilities. To achieve this vision, AI agents must effectively plan, perform multi-step reasoning and actions, respond to novel observations, and recover from errors, to successfully complete complex tasks across a wide range of scenarios. In this work, we introduce Magentic-One, a high-performing open-source agentic system for solving such tasks. Magentic-One uses a multi-agent architecture where a lead agent, the Orchestrator, plans, tracks progress, and re-plans to recover from errors. Throughout task execution, the Orchestrator directs other specialized agents to perform tasks as needed, such as operating a web browser, navigating local files, or writing and executing Python code. We show that Magentic-One achieves statistically competitive performance to the state-of-the-art on three diverse and challenging agentic benchmarks: GAIA, AssistantBench, and WebArena. Magentic-One achieves these results without modification to core agent capabilities or to how they collaborate, demonstrating progress towards generalist agentic systems. Moreover, Magentic-One's modular design allows agents to be added or removed from the team without additional prompt tuning or training, easing development and making it extensible to future scenarios. We provide an open-source implementation of Magentic-One, and we include AutoGenBench, a standalone tool for agentic evaluation. AutoGenBench provides built-in controls for repetition and isolation to run agentic benchmarks in a rigorous and contained manner -- which is important when agents' actions have side-effects. Magentic-One, AutoGenBench and detailed empirical performance evaluations of Magentic-One, including ablations and error analysis are available at https://aka.ms/magentic-one

Motivation & Objective

• Motivate the development of generalist agentic systems capable of solving complex tasks across diverse domains.
• Propose a modular multi-agent architecture where an Orchestrator coordinates specialized agents (WebSurfer, FileSurfer, Coder, ComputerTerminal).
• Introduce AutoGenBench for rigorous, repeatable evaluation of agentic systems in stateful environments.
• Demonstrate that a non-modified, modular multi-agent setup can match state-of-the-art performance on multiple benchmarks.
• Highlight extensibility, ease of adding/removing agents, and the potential for future scenario adaptation.

Proposed method

• Propose Magentic-One’s two-loop orchestration: an outer loop maintaining a task ledger (plan, facts, educated guesses) and an inner loop maintaining a progress ledger (task assignments and progress tracking).
• Use an Orchestrator to generate step-by-step plans and allocate subtasks to specialized agents, with reset/replanning to recover from stalling or errors.
• Deploy a team of specialized agents (WebSurfer for browser/web tasks, FileSurfer for file interactions, Coder for code/artifact creation, ComputerTerminal for code execution) under a shared control flow.
• Incorporate a recovery mechanism via a counter-based inner loop and a reflection step when stuck, enabling limited error recovery.
• Introduce AutoGenBench as a standalone, controlled evaluation framework to ensure isolated, repeatable benchmarking for agentic systems.
• Evaluate across three benchmarks (GAIA, AssistantBench, WebArena) with open-source implementation and ablation analyses.

Figure 1 : An illustration of the Magentic-One mutli-agent team completing a complex task from the GAIA benchmark. Magentic-One’s Orchestrator agent creates a plan, delegates tasks to other agents, and tracks progress towards the goal, dynamically revising the plan as needed. The Orchestrator can de

Experimental results

Research questions

• RQ1 Can Magentic-One achieve competitive task completion or accuracy across diverse agentic benchmarks (GAIA, AssistantBench, WebArena)?
• RQ2 How does the modular multi-agent design (Orchestrator plus specialized agents) contribute to performance and generalization without prompt training or core architecture changes?
• RQ3 What is the impact of the evaluation framework (AutoGenBench) on reproducibility, safety, and variance in agentic tasks?
• RQ4 What are the individual contributions of each agent to overall performance, and where are the main failure modes?
• RQ5 Does the system maintain performance when varying underlying LLMs (e.g., GPT-4o vs o1-preview)?

Key findings

• Magentic-One achieves task completion rates of 38% on GAIA, 32.8% on WebArena, and 27.7% on AssistantBench, with GPT-4o and o1 configurations showing statistically competitive performance to SOTA baselines.
• Using GPT-4o alone, Magentic-One attains 32.33% (GAIA), 11.0% (AssistantBench EM), 25.3% (AssistantBench accuracy), and 32.8% (WebArena) in the reported setup; with GPT-4o and o1, GAIA improves to 38.00%, AssistantBench 13.3% (EM) and 27.7% (accuracy).
• Ablation analyses indicate additive value of each agent to overall performance and reveal clear error modes, guiding opportunities for improvement.
• AutoGenBench enables controlled, repeatable benchmarking by ensuring fixed initial conditions, isolation between runs, and safe evaluation of stateful tasks.
• The results demonstrate the feasibility of a generalist agentic system that can operate across web and file-based tasks without core changes to agent capabilities or collaboration schemes.

Figure 2 : Magentic-One features an Orchestrator agent that implements two loops: an outer loop and an inner loop. The outer loop (lighter background with solid arrows) manages the task ledger (containing facts, guesses, and plan). The inner loop (darker background with dotted arrows) manages the pr