[Paper Review] A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization

Sungmin Kang, Gabin An+1 more|arXiv (Cornell University)|Aug 10, 2023

Software Engineering Research46 references9 citations

TL;DR

AutoFL uses OpenAI function-calling to let an LLM explore a code repository and produce root-cause explanations plus fault-location predictions from a single failing test, achieving state-of-the-art acc@1 on Defects4J compared to standalone techniques.

ABSTRACT

Fault Localization (FL), in which a developer seeks to identify which part of the code is malfunctioning and needs to be fixed, is a recurring challenge in debugging. To reduce developer burden, many automated FL techniques have been proposed. However, prior work has noted that existing techniques fail to provide rationales for the suggested locations, hindering developer adoption of these techniques. With this in mind, we propose AutoFL, a Large Language Model (LLM)-based FL technique that generates an explanation of the bug along with a suggested fault location. AutoFL prompts an LLM to use function calls to navigate a repository, so that it can effectively localize faults over a large software repository and overcome the limit of the LLM context length. Extensive experiments on 798 real-world bugs in Java and Python reveal AutoFL improves method-level acc@1 by up to 233.3% over baselines. Furthermore, developers were interviewed on their impression of AutoFL-generated explanations, showing that developers generally liked the natural language explanations of AutoFL, and that they preferred reading a few, high-quality explanations instead of many.

Motivation & Objective

• Motivate the use of large language models for fault localization to improve performance and provide human-understandable explanations.
• Develop AutoFL, an automated fault localization pipeline that uses LLMs with function-calling to explore large codebases from a single failing test.
• Enable staged LLM interaction to generate root-cause explanations and precise fault-location outputs.
• Propose a postprocessing pipeline to map LLM outputs to actual code elements and rank fault candidates.

Proposed method

• Provide an LLM-driven fault localization pipeline AutoFL that uses function calls to inspect covered classes and methods, and retrieve code snippets and Javadoc.
• Use a two-stage prompting process: Stage 1 generates a root-cause explanation; Stage 2 outputs the most likely culprit method signatures.
• Implement a postprocessing step that aggregates multiple AutoFL runs and ranks suspected methods by a weighted score.
• Evaluate AutoFL on Defects4J with gpt-3.5-turbo-0613, comparing against SBFL, MBFL, IRFL, and LLM-based baselines.
• Incorporate a repeat-and-aggregate strategy (R=5) to improve robustness and ranking accuracy.

Experimental results

Research questions

• RQ1 Can an LLM with controlled tool use identify the root cause of a bug from a failing test?
• RQ2 How well does AutoFL locate the true buggy method compared with traditional fault localization techniques under failing-test-only conditions?
• RQ3 Does postprocessing and run aggregation improve fault localization accuracy and ranking stability?
• RQ4 What are typical LLM function-call patterns that lead to successful versus unsuccessful fault localization?
• RQ5 What are the limitations and failure modes of AutoFL on real-world bugs?

Key findings

• AutoFL locates the true buggy method on the first guess (acc@1) in 149 of 353 Defects4J cases, outperforming all standalone techniques evaluated.
• AutoFL outperforms the SBFL baseline when restricted to failing tests, showing substantial improvement in acc@1 (up to 338% in certain comparisons).
• Postprocessing and run aggregation substantially improve performance; merging results across runs yields higher acc@k across the board and reduces false positives when consensus is strong.
• More than 40% of bugs AutoFL localizes on its first rank are not correctly localized by the best SBFL baseline, indicating orthogonality of AutoFL to traditional FL methods.
• There exist failure cases (e.g., Time-25) where AutoFL provides shallow analysis and misses deeper bug causes, highlighting areas for future enhancement.
• Across runs, the average number of function calls is about 5.36 with variability, and successful patterns often involve successive examination of covered classes and methods before inspecting code.