[Paper Review] Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applications

Raphael Shu, Nilaksh Das+3 more|arXiv (Cornell University)|Dec 6, 2024

Collaboration in agile enterprisesBusiness, Management and Accounting3 citations

TL;DR

This paper proposes a hierarchical multi-agent collaboration framework using LLM-powered agents for enterprise applications, enabling coordinated problem-solving through structured task delegation and optimized communication. It achieves 90% goal success rate in complex domains like software development and mortgage processing, with payload referencing boosting code task performance by 23% and reducing communication overhead by 27%.

ABSTRACT

AI agents powered by large language models (LLMs) have shown strong capabilities in problem solving. Through combining many intelligent agents, multi-agent collaboration has emerged as a promising approach to tackle complex, multi-faceted problems that exceed the capabilities of single AI agents. However, designing the collaboration protocols and evaluating the effectiveness of these systems remains a significant challenge, especially for enterprise applications. This report addresses these challenges by presenting a comprehensive evaluation of coordination and routing capabilities in a novel multi-agent collaboration framework. We evaluate two key operational modes: (1) a coordination mode enabling complex task completion through parallel communication and payload referencing, and (2) a routing mode for efficient message forwarding between agents. We benchmark on a set of handcrafted scenarios from three enterprise domains, which are publicly released with the report. For coordination capabilities, we demonstrate the effectiveness of inter-agent communication and payload referencing mechanisms, achieving end-to-end goal success rates of 90%. Our analysis yields several key findings: multi-agent collaboration enhances goal success rates by up to 70% compared to single-agent approaches in our benchmarks; payload referencing improves performance on code-intensive tasks by 23%; latency can be substantially reduced with a routing mechanism that selectively bypasses agent orchestration. These findings offer valuable guidance for enterprise deployments of multi-agent systems and advance the development of scalable, efficient multi-agent collaboration frameworks.

Motivation & Objective

• Address the challenge of designing effective collaboration protocols for multi-agent systems in enterprise environments.
• Improve goal success rates in complex, multi-faceted problems beyond the reach of single agents.
• Evaluate coordination and routing mechanisms for efficiency, scalability, and real-world applicability.
• Develop a benchmarking framework with handcrafted enterprise scenarios for reliable evaluation of multi-agent systems.
• Optimize inter-agent communication to reduce latency and overhead, especially in code-intensive workloads.

Proposed method

• Proposes a hierarchical agent architecture with a central supervisor agent managing task planning, delegation, and coordination.
• Employs a coordination mode enabling parallel communication and payload referencing between agents to share large content blocks like code snippets.
• Introduces a routing mode that selectively bypasses orchestration to reduce latency by avoiding unnecessary agent hops.
• Uses assertion-based benchmarking with automated LLM-based evaluation to assess goal success and consistency.
• Employs Claude 3.5 Sonnet (20241022) models for both supervisor and specialist agents across three enterprise domains: Travel, Mortgage, and Software Development.
• Releases a public benchmark dataset with handcrafted scenarios to enable reproducibility and further research.

Figure 1: Illustration of the hierarchical agents approach for multi-agent collaboration. In a centralized hierarchy, a supervisor agent oversees and assigns tasks to specialist agents. The figure demonstrates a multi-layer hierarchy, where an agent can function as both a specialist agent and a supe

Experimental results

Research questions

• RQ1 How can a hierarchical multi-agent collaboration framework effectively coordinate specialized agents to solve complex enterprise problems?
• RQ2 What impact do communication mechanisms—particularly payload referencing—have on performance and efficiency in code-intensive tasks?
• RQ3 How does multi-agent collaboration compare to single-agent approaches in terms of goal success rate across diverse enterprise domains?
• RQ4 To what extent can routing optimizations reduce latency in multi-agent coordination without sacrificing reliability?
• RQ5 How effective is an automated LLM-based evaluation framework in measuring goal success compared to human judgment?

Key findings

• Multi-agent collaboration improved goal success rates by up to 70% compared to single-agent approaches in the benchmarked scenarios.
• The payload referencing mechanism increased goal success rate by 23% in code-intensive tasks and reduced average communication overhead per turn by 27%.
• The system achieved an overall goal success rate of 90% across all evaluated enterprise domains when using Claude 3.5 Sonnet models.
• A routing mechanism that selectively bypasses agent orchestration significantly reduced latency, especially in complex workflows.
• The automated LLM-based evaluation framework showed high agreement (validated by human judgment) with human-annotated success rates, enabling scalable and reliable benchmarking.
• The framework demonstrated strong performance in the Software Development domain, where single-agent approaches regressed by up to 37% in goal success rate.

Figure 2: Example of parallel agent communication. In this example, the supervisor agent simultaneously communicates with multiple agents as the tasks can be completed independently.