[论文解读] A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization

Sungmin Kang, Gabin An+1 more|arXiv (Cornell University)|Aug 10, 2023

Software Engineering Research参考文献 46被引用 9

一句话总结

AutoFL 使用 OpenAI 函数调用让大语言模型（LLM）探索代码仓库，并从单次失败测试中给出根因解释和故障定位预测，在 Defects4J 上的 acc@1 达到最先进水平，优于独立技术。

ABSTRACT

Fault Localization (FL), in which a developer seeks to identify which part of the code is malfunctioning and needs to be fixed, is a recurring challenge in debugging. To reduce developer burden, many automated FL techniques have been proposed. However, prior work has noted that existing techniques fail to provide rationales for the suggested locations, hindering developer adoption of these techniques. With this in mind, we propose AutoFL, a Large Language Model (LLM)-based FL technique that generates an explanation of the bug along with a suggested fault location. AutoFL prompts an LLM to use function calls to navigate a repository, so that it can effectively localize faults over a large software repository and overcome the limit of the LLM context length. Extensive experiments on 798 real-world bugs in Java and Python reveal AutoFL improves method-level acc@1 by up to 233.3% over baselines. Furthermore, developers were interviewed on their impression of AutoFL-generated explanations, showing that developers generally liked the natural language explanations of AutoFL, and that they preferred reading a few, high-quality explanations instead of many.

研究动机与目标

• Motivate the use of large language models for fault localization to improve performance and provide human-understandable explanations.
• Develop AutoFL, an automated fault localization pipeline that uses LLMs with function-calling to explore large codebases from a single failing test.
• Enable staged LLM interaction to generate root-cause explanations and precise fault-location outputs.
• Propose a postprocessing pipeline to map LLM outputs to actual code elements and rank fault candidates.

提出的方法

• Provide an LLM-driven fault localization pipeline AutoFL that uses function calls to inspect covered classes and methods, and retrieve code snippets and Javadoc.
• Use a two-stage prompting process: Stage 1 generates a root-cause explanation; Stage 2 outputs the most likely culprit method signatures.
• Implement a postprocessing step that aggregates multiple AutoFL runs and ranks suspected methods by a weighted score.
• Evaluate AutoFL on Defects4J with gpt-3.5-turbo-0613, comparing against SBFL, MBFL, IRFL, and LLM-based baselines.
• Incorporate a repeat-and-aggregate strategy (R=5) to improve robustness and ranking accuracy.

实验结果

研究问题

• RQ1 Can an LLM with controlled tool use identify the root cause of a bug from a failing test?
• RQ2 How well does AutoFL locate the true buggy method compared with traditional fault localization techniques under failing-test-only conditions?
• RQ3 Does postprocessing and run aggregation improve fault localization accuracy and ranking stability?
• RQ4 What are typical LLM function-call patterns that lead to successful versus unsuccessful fault localization?
• RQ5 What are the limitations and failure modes of AutoFL on real-world bugs?

主要发现

• AutoFL locates the true buggy method on the first guess (acc@1) in 149 of 353 Defects4J cases, outperforming all standalone techniques evaluated.
• AutoFL outperforms the SBFL baseline when restricted to failing tests, showing substantial improvement in acc@1 (up to 338% in certain comparisons).
• Postprocessing and run aggregation substantially improve performance; merging results across runs yields higher acc@k across the board and reduces false positives when consensus is strong.
• More than 40% of bugs AutoFL localizes on its first rank are not correctly localized by the best SBFL baseline, indicating orthogonality of AutoFL to traditional FL methods.
• There exist failure cases (e.g., Time-25) where AutoFL provides shallow analysis and misses deeper bug causes, highlighting areas for future enhancement.
• Across runs, the average number of function calls is about 5.36 with variability, and successful patterns often involve successive examination of covered classes and methods before inspecting code.