[论文解读] Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applications

Raphael Shu, Nilaksh Das+3 more|arXiv (Cornell University)|Dec 6, 2024

Collaboration in agile enterprisesBusiness, Management and Accounting被引用 3

一句话总结

本文提出了一种基于大语言模型驱动的代理的分层多代理协作框架，用于企业应用，通过结构化的任务委派和优化的通信实现协调式问题求解。在软件开发和抵押贷款处理等复杂领域，该框架实现了90%的目标成功率，其中通过负载引用机制使代码任务性能提升23%，通信开销减少27%。

ABSTRACT

AI agents powered by large language models (LLMs) have shown strong capabilities in problem solving. Through combining many intelligent agents, multi-agent collaboration has emerged as a promising approach to tackle complex, multi-faceted problems that exceed the capabilities of single AI agents. However, designing the collaboration protocols and evaluating the effectiveness of these systems remains a significant challenge, especially for enterprise applications. This report addresses these challenges by presenting a comprehensive evaluation of coordination and routing capabilities in a novel multi-agent collaboration framework. We evaluate two key operational modes: (1) a coordination mode enabling complex task completion through parallel communication and payload referencing, and (2) a routing mode for efficient message forwarding between agents. We benchmark on a set of handcrafted scenarios from three enterprise domains, which are publicly released with the report. For coordination capabilities, we demonstrate the effectiveness of inter-agent communication and payload referencing mechanisms, achieving end-to-end goal success rates of 90%. Our analysis yields several key findings: multi-agent collaboration enhances goal success rates by up to 70% compared to single-agent approaches in our benchmarks; payload referencing improves performance on code-intensive tasks by 23%; latency can be substantially reduced with a routing mechanism that selectively bypasses agent orchestration. These findings offer valuable guidance for enterprise deployments of multi-agent systems and advance the development of scalable, efficient multi-agent collaboration frameworks.

研究动机与目标

• 解决在企业环境中设计高效多代理协作协议的挑战。
• 提升在单个代理难以应对的复杂多维度问题中的目标成功率。
• 评估协调与路由机制在效率、可扩展性及现实适用性方面的表现。
• 构建一个包含手工设计的企业场景的基准测试框架，以实现对多代理系统的可靠评估。
• 优化代理间通信，降低延迟与开销，特别是在代码密集型工作负载中。

提出的方法

• 提出一种分层代理架构，由中央协调代理负责任务规划、委派与协调。
• 采用一种协调模式，支持代理间并行通信与负载引用，以共享大型内容块（如代码片段）。
• 引入一种路由模式，通过选择性跳过编排流程来减少延迟，避免不必要的代理跳转。
• 采用基于断言的基准测试方法，结合自动化的LLM评估，以衡量目标达成率与一致性。
• 在三个企业领域（旅行、抵押贷款、软件开发）中，均使用Claude 3.5 Sonnet（20241022）模型作为协调代理与专业代理。
• 发布一个公开的基准数据集，包含手工设计的场景，以支持可复现性与进一步研究。

Figure 1: Illustration of the hierarchical agents approach for multi-agent collaboration. In a centralized hierarchy, a supervisor agent oversees and assigns tasks to specialist agents. The figure demonstrates a multi-layer hierarchy, where an agent can function as both a specialist agent and a supe

实验结果

研究问题

• RQ1 如何通过分层多代理协作框架有效协调专业化代理，以解决复杂的企业问题？
• RQ2 在代码密集型任务中，通信机制（尤其是负载引用）对性能与效率有何影响？
• RQ3 在多样化的企业领域中，多代理协作与单代理方法相比，目标成功率有何差异？
• RQ4 在不牺牲可靠性的前提下，路由优化能在多大程度上降低多代理协调中的延迟？
• RQ5 与人工判断相比，自动化的LLM评估框架在衡量目标达成率方面的有效性如何？

主要发现

• 与单代理方法相比，多代理协作在基准测试场景中使目标成功率最高提升了70%。
• 负载引用机制使代码密集型任务的目标成功率提升23%，并使每轮通信的平均开销降低27%。
• 在使用Claude 3.5 Sonnet模型时，该系统在所有评估的企业领域中实现了90%的整体目标成功率。
• 一种选择性跳过代理编排的路由机制显著降低了延迟，尤其在复杂工作流中表现突出。
• 自动化的LLM评估框架与人工标注的成功率高度一致（经人工判断验证），实现了可扩展且可靠的基准测试。
• 该框架在软件开发领域表现出色，而单代理方法在该领域中的目标成功率最高下降了37%。

Figure 2: Example of parallel agent communication. In this example, the supervisor agent simultaneously communicates with multiple agents as the tasks can be completed independently.