[Paper Review] AgentFL: Scaling LLM-based Fault Localization to Project-Level Context

Yihao Qin, Shangwen Wang+5 more|arXiv (Cornell University)|Mar 25, 2024

Software Engineering Research4 citations

TL;DR

AgentFL is a multi-agent LLM system that scales fault localization to project-level codebases by decomposing the process into three stages: fault comprehension, codebase navigation, and fault confirmation. It achieves state-of-the-art performance by using specialized agents with tools like test behavior tracking and document-guided search, localizing 157 out of 395 bugs in Top-1 with low cost (avg. $0.074 and 97 seconds per bug).

ABSTRACT

Fault Localization (FL) is an essential step during the debugging process. With the strong capabilities of code comprehension, the recent Large Language Models (LLMs) have demonstrated promising performance in diagnosing bugs in the code. Nevertheless, due to LLMs' limited performance in handling long contexts, existing LLM-based fault localization remains on localizing bugs within a small code scope (i.e., a method or a class), which struggles to diagnose bugs for a large code scope (i.e., an entire software system). To address the limitation, this paper presents AgentFL, a multi-agent system based on ChatGPT for automated fault localization. By simulating the behavior of a human developer, AgentFL models the FL task as a three-step process, which involves comprehension, navigation, and confirmation. Within each step, AgentFL hires agents with diversified expertise, each of which utilizes different tools to handle specific tasks. Particularly, we adopt a series of auxiliary strategies such as Test Behavior Tracking, Document-Guided Search, and Multi-Round Dialogue to overcome the challenges in each step. The evaluation on the widely used Defects4J-V1.2.0 benchmark shows that AgentFL can localize 157 out of 395 bugs within Top-1, which outperforms the other LLM-based approaches and exhibits complementarity to the state-of-the-art learning-based techniques. Additionally, we confirm the indispensability of the components in AgentFL with the ablation study and demonstrate the usability of AgentFL through a user study. Finally, the cost analysis shows that AgentFL spends an average of only 0.074 dollars and 97 seconds for a single bug.

Motivation & Objective

• Address the limitation of existing LLM-based fault localization methods, which are restricted to small code scopes (e.g., single methods) due to context length constraints.
• Enable project-level fault localization by simulating human developer debugging behaviors—comprehending faults, navigating codebases, and confirming buggy locations.
• Overcome LLMs’ performance degradation on long contexts by decomposing the task into controlled, multi-step phases with specialized agents.
• Improve accuracy and usability of LLM-based fault localization through auxiliary strategies like test behavior tracking and multi-round dialogue.
• Demonstrate cost-effectiveness and practical usability through cost analysis and a user study.

Proposed method

• Decompose fault localization into three stages: Fault Comprehension (analyzing failure traces), Codebase Navigation (exploring relevant code paths), and Fault Confirmation (validating suspected buggy locations).
• Deploy specialized LLM-driven agents for each stage, each equipped with tailored tools (e.g., test execution, code search, semantic similarity matching).
• Integrate Test Behavior Tracking to trace execution paths of failing tests and guide agent navigation toward relevant code.
• Apply Document-Guided Search to retrieve semantically relevant code snippets from the project using embeddings and RAG-like retrieval.
• Use Multi-Round Dialogue to refine agent reasoning and improve prediction accuracy through iterative clarification and validation.
• Leverage GPT-3.5-turbo-16k-0613 with prompt engineering and input filtering to avoid data leakage while maintaining performance.

Experimental results

Research questions

• RQ1 Can a multi-agent LLM system effectively scale fault localization from method-level to project-level codebases?
• RQ2 How does AgentFL’s three-stage architecture (comprehension, navigation, confirmation) improve localization accuracy compared to baseline LLM prompting?
• RQ3 What is the impact of auxiliary strategies—Test Behavior Tracking, Document-Guided Search, and Multi-Round Dialogue—on fault localization performance?
• RQ4 How usable is AgentFL in practice, and does it provide meaningful explanations that assist developers in identifying bugs?
• RQ5 What are the computational and financial costs of deploying AgentFL at scale?

Key findings

• AgentFL localizes 157 out of 395 bugs in Defects4J-V1.2.0 within the Top-1 prediction, outperforming existing LLM-based approaches.
• The system demonstrates complementarity with state-of-the-art learning-based techniques, suggesting potential for hybrid fault localization pipelines.
• In a user study, participants found AgentFL’s explanations helpful, with over 80% of failures involving fewer than three incorrect predictions before the correct one.
• AgentFL achieves low cost and time consumption, averaging $0.074 and 97 seconds per bug, with 95% of cases under $0.20 and 200 seconds.
• Ablation studies confirm the indispensability of core components like Test Behavior Tracking and Multi-Round Dialogue for optimal performance.
• Case studies show that even when the top prediction is incorrect, AgentFL often identifies the correct root cause and suggests relevant methods (e.g., callers), reducing developer effort.