[Paper Review] Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization

Yoïchi Ishibashi, Yoshimasa Nishimura|arXiv (Cornell University)|Apr 2, 2024

Modular Robots and Swarm Intelligence4 citations

TL;DR

This paper proposes Self-Organized Agents (SoA), a novel LLM multi-agent framework that enables scalable, dynamic code generation and optimization by allowing agents to autonomously multiply based on problem complexity. Each agent manages a fixed, small code segment, while collaborative self-organization allows the system to scale to ultra-large codebases—achieving a 5% improvement in Pass@1 accuracy over state-of-the-art single-agent baselines on HumanEval.

ABSTRACT

Recent advancements in automatic code generation using large language model (LLM) agent have brought us closer to the future of automated software development. However, existing single-agent approaches face limitations in generating and improving large-scale, complex codebases due to constraints in context length. To tackle this challenge, we propose Self-Organized multi-Agent framework (SoA), a novel multi-agent framework that enables the scalable and efficient generation and optimization of large-scale code. In SoA, self-organized agents operate independently to generate and modify code components while seamlessly collaborating to construct the overall codebase. A key feature of our framework is the automatic multiplication of agents based on problem complexity, allowing for dynamic scalability. This enables the overall code volume to be increased indefinitely according to the number of agents, while the amount of code managed by each agent remains constant. We evaluate SoA on the HumanEval benchmark and demonstrate that, compared to a single-agent system, each agent in SoA handles significantly less code, yet the overall generated code is substantially greater. Moreover, SoA surpasses the powerful single-agent baseline by 5% in terms of Pass@1 accuracy.

Motivation & Objective

• To overcome the scalability limitations of single-agent LLMs in generating and optimizing large, complex codebases due to context length constraints.
• To enable dynamic, self-organized code generation by allowing agents to autonomously multiply based on problem complexity.
• To maintain constant code management per agent while enabling indefinite growth of the overall codebase.
• To improve code generation quality and efficiency through decentralized, collaborative agent architecture.
• To provide a flexible, micro-level framework for code generation that complements existing macro-level multi-agent software development approaches.

Proposed method

• Agents are assigned to generate and modify specific code components based on functional decomposition rather than predefined roles.
• The framework employs self-organization, where agents coordinate and collaborate without centralized control to assemble the full codebase.
• Agent multiplication is triggered automatically based on problem complexity, ensuring each agent handles a constant, manageable code volume.
• A tree-like structure with agents as nodes enables scalable, parallel code generation and modification.
• The system uses prompt engineering techniques such as Tree-of-Thought (ToT) and Skeleton of Thought (SoT) to guide reasoning and improve generation efficiency.
• Agents communicate and synchronize through a lightweight coordination mechanism to maintain consistency and correctness across the codebase.

Figure 1: Left (single agent): A single agent is solely responsible for the entire implementation. As the codebase grows larger, the load increases for code generation, modification, and memory management, making it difficult to manage and develop. The larger the entire codebase becomes, the more it

Experimental results

Research questions

• RQ1 Can a self-organizing multi-agent framework dynamically scale code generation capacity while keeping individual agent workloads constant?
• RQ2 How does agent multiplication based on problem complexity affect overall code volume and generation quality?
• RQ3 To what extent does the SoA framework outperform single-agent baselines in code generation accuracy and scalability?
• RQ4 How does the absence of fixed roles improve adaptability and performance in complex code generation tasks?
• RQ5 What is the impact of decentralized, collaborative agent behavior on code quality and system robustness?

Key findings

• SoA achieved a 5% higher Pass@1 accuracy on the HumanEval benchmark compared to the Reflexion single-agent baseline.
• Each agent in SoA managed significantly less code than a single-agent system, while the total generated code volume was substantially larger.
• The framework demonstrated dynamic scalability through automatic agent multiplication, enabling indefinite growth of the codebase.
• Agent self-organization allowed for efficient collaboration without centralized oversight, maintaining code consistency and correctness.
• The micro-level, function-based agent allocation outperformed role-based multi-agent frameworks in adaptability and code generation quality.
• The system's performance was sensitive to LLM choice and test quality, indicating room for optimization in communication and fault tolerance.

Figure 2: Overview of code generation. Child agents generate executable Python function from a given docstring. The Mother agent generates the skeleton of the function. The Mother spawns a new initialized agent (Child or Mother) and delegates unimplemented functions.