[Paper Review] MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

Sirui Hong, Zhuge, Mingchen+13 more|arXiv (Cornell University)|Aug 1, 2023

Multi-Agent Systems and NegotiationComputer Science130 citations

TL;DR

MetaGPT introduces a meta-programming framework that uses SOP-driven, structured outputs and executable feedback to coordinate multi-agent LLM collaboration, achieving state-of-the-art code generation and robust software development in benchmarks.

ABSTRACT

Remarkable progress has been made on automated problem solving through societies of agents based on large language models (LLMs). Existing LLM-based multi-agent systems can already solve simple dialogue tasks. Solutions to more complex tasks, however, are complicated through logic inconsistencies due to cascading hallucinations caused by naively chaining LLMs. Here we introduce MetaGPT, an innovative meta-programming framework incorporating efficient human workflows into LLM-based multi-agent collaborations. MetaGPT encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. MetaGPT utilizes an assembly line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks into subtasks involving many agents working together. On collaborative software engineering benchmarks, MetaGPT generates more coherent solutions than previous chat-based multi-agent systems. Our project can be found at https://github.com/geekan/MetaGPT

Motivation & Objective

• Motivate improving coherence and correctness in LLM-based multi-agent problem solving by incorporating standardized operating procedures (SOPs).
• Decompose complex software tasks into roles and workflows to reduce cascading hallucinations across agents.
• Enable structured outputs (documents/diagrams) and a publish-subscribe information flow to improve communication efficiency.
• Introduce an executable feedback mechanism to debug and run code during runtime for higher-quality code generation.

Proposed method

• Define five specialized agent roles (Product Manager, Architect, Project Manager, Engineer, QA Engineer) with task-specific profiles and constraints.
• Implement a structured, document-based communication protocol with a shared message pool and a subscription mechanism to reduce information overload.
• Adopt an SOP-driven software development workflow that serializes task progression from requirements to design, implementation, and testing.
• Introduce executable feedback where the Engineer runs unit tests and iteratively debugs code within a capped retry limit (up to 3 retries).
• Evaluate on HumanEval, MBPP, and a new SoftwareDev benchmark using Pass@k metrics and human/system-level evaluations, comparing against AutoGPT, LangChain, AgentVerse, and ChatDev.

Figure 1: The software development SOPs between MetaGPT and real-world human teams. In software engineering, SOPs promote collaboration among various roles. MetaGPT showcases its ability to decompose complex tasks into specific actionable procedures assigned to various roles (e.g., Product Manager,

Experimental results

Research questions

• RQ1 How does incorporating SOPs and role specialization affect coherence and error rates in multi-agent code generation?
• RQ2 Can a publish-subscribe, structured-output communication protocol improve task efficiency and reduce hallucinations in LLM-based collaboration?
• RQ3 Does executable feedback during runtime significantly improve code quality and executability on standard benchmarks?
• RQ4 How does MetaGPT perform on standard code-generation benchmarks compared to existing multi-agent frameworks and generic LLMs?

Key findings

• MetaGPT achieves state-of-the-art Pass@1 scores on HumanEval and MBPP, reaching 85.9% and 87.7% respectively.
• MetaGPT attains a 100% task completion rate on its experiments.
• On the SoftwareDev benchmark, MetaGPT outperforms ChatDev across most metrics, with an executability score of 3.75 and shorter running time (503s).
• MetaGPT uses more tokens overall (24,613 or 31,255) but achieves lower human revision cost (0.83) and higher executability with structured SOPs.
• Executable feedback yields a 4.2% and 5.4% absolute improvement in Pass@1 on HumanEval and MBPP, respectively.

Figure 2: An example of the communication protocol (left) and iterative programming with executable feedback (right). Left : Agents use a shared message pool to publish structured messages. They can also subscribe to relevant messages based on their profiles. Right : After generating the initial cod