[Paper Review] MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution

Wei Tao, Yucheng Zhou+4 more|arXiv (Cornell University)|Mar 26, 2024

Scientific Computing and Data Management8 citations

TL;DR

MAGIS uses a four-agent framework (Manager, Repository Custodian, Developer, QA Engineer) to coordinate LLMs for resolving GitHub issues, achieving substantially higher issue resolution on SWE-bench than strong baselines.

ABSTRACT

In software development, resolving the emergent issues within GitHub repositories is a complex challenge that involves not only the incorporation of new code but also the maintenance of existing code. Large Language Models (LLMs) have shown promise in code generation but face difficulties in resolving Github issues, particularly at the repository level. To overcome this challenge, we empirically study the reason why LLMs fail to resolve GitHub issues and analyze the major factors. Motivated by the empirical findings, we propose a novel LLM-based Multi-Agent framework for GitHub Issue reSolution, MAGIS, consisting of four agents customized for software evolution: Manager, Repository Custodian, Developer, and Quality Assurance Engineer agents. This framework leverages the collaboration of various agents in the planning and coding process to unlock the potential of LLMs to resolve GitHub issues. In experiments, we employ the SWE-bench benchmark to compare MAGIS with popular LLMs, including GPT-3.5, GPT-4, and Claude-2. MAGIS can resolve 13.94% GitHub issues, significantly outperforming the baselines. Specifically, MAGIS achieves an eight-fold increase in resolved ratio over the direct application of GPT-4, the advanced LLM.

Motivation & Objective

• Investigate why LLMs struggle with repository-level GitHub issue resolution and identify impact factors.
• Propose MAGIS, an LLM-based multi-agent framework for coordinated issue resolution in software repositories.
• Evaluate MAGIS against strong LLM baselines on SWE-bench across with-Oracle and without-Oracle settings.
• Analyze planning and coding factors that influence GitHub issue resolution success.

Proposed method

• Empirical analysis of LLM performance under with-Oracle and without-Oracle settings to identify factors like line location and code-change complexity.
• Design of MAGIS with four agent types and their collaborative planning and coding workflow.
• Development of algorithms for locating relevant files, team formation, and kick-off planning.
• Iterative code modification and QA review guided by LLM prompts to generate repository-level changes.
• Evaluation against GPT-3.5, GPT-4, Claude-2 and SWE-Llama baselines on SWE-bench using applied and resolved ratios.

Experimental results

Research questions

• RQ1 What factors affect LLM performance in with-Oracle GitHub issue resolution (e.g., line location, code-change complexity)?
• RQ2 What factors affect LLM performance in without-Oracle resolution (e.g., file retrieval/recall)?
• RQ3 How does a multi-agent MAGIS framework improve GitHub issue resolution compared to single-LLM baselines?
• RQ4 What is the impact of planning and QA processes on resolution effectiveness?

Key findings

• MAGIS achieves a 13.94% resolved ratio, eight times higher than GPT-4 baseline performance.
• MAGIS substantially outperforms GPT-4 and Claude-2 in both applied and resolved metrics on SWE-bench.
• Line location accuracy correlates positively with resolution probability, especially for Claude-2.
• Code-change complexity negatively correlates with resolution across GPT-3.5, GPT-4, and Claude-2.
• A planning and QA-enabled MAGIS variant shows notable gains; without QA or hints, resolution remains higher than GPT-4 but lower than full MAGIS.
• Ablation studies indicate QA and human hints further improve performance.