[论文解读] An Empirical Study on the Potential of LLMs in Automated Software Refactoring

Bo Liu, Yanjie Jiang+4 more|arXiv (Cornell University)|Nov 7, 2024

Digital Rights Management and Security被引用 4

一句话总结

本文评估了大型语言模型（LLMs）如ChatGPT和Gemini在自动化软件重构中的潜力，发现通过优化提示工程，LLMs能够识别86.7%的重构机会，并推荐与人类专家相当的高质量解决方案。为缓解因幻觉导致的语法或功能错误，作者提出RefactoringMirror——一种检测并重新应用的策略，通过经验证的重构引擎重新应用LLM建议的重构，成功避免了所有不安全的更改。

ABSTRACT

Recent advances in large language models (LLMs), make it potentially feasible to automatically refactor source code with LLMs. However, it remains unclear how well LLMs perform compared to human experts in conducting refactorings automatically and accurately. To fill this gap, in this paper, we conduct an empirical study to investigate the potential of LLMs in automated software refactoring, focusing on the identification of refactoring opportunities and the recommendation of refactoring solutions. We first construct a high-quality refactoring dataset comprising 180 real-world refactorings from 20 projects, and conduct the empirical study on the dataset. With the to-be-refactored Java documents as input, ChatGPT and Gemini identified only 28 and 7 respectively out of the 180 refactoring opportunities. However, explaining the expected refactoring subcategories and narrowing the search space in the prompts substantially increased the success rate of ChatGPT from 15.6% to 86.7%. Concerning the recommendation of refactoring solutions, ChatGPT recommended 176 refactoring solutions for the 180 refactorings, and 63.6% of the recommended solutions were comparable to (even better than) those constructed by human experts. However, 13 out of the 176 solutions suggested by ChatGPT and 9 out of the 137 solutions suggested by Gemini were unsafe in that they either changed the functionality of the source code or introduced syntax errors, which indicate the risk of LLM-based refactoring. To this end, we propose a detect-and-reapply tactic, called RefactoringMirror, to avoid such unsafe refactorings. By reapplying the identified refactorings to the original code using thoroughly tested refactoring engines, we can effectively mitigate the risks associated with LLM-based automated refactoring while still leveraging LLM's intelligence to obtain valuable refactoring recommendations.

研究动机与目标

• 调查LLMs是否能够准确识别生产环境中Java代码的真实重构机会。
• 评估LLMs推荐的重构解决方案质量与人类专家相比如何。
• 解决LLM生成的重构可能引入语法错误或改变功能的风险。
• 提出并验证一种检测并重新应用的策略，以确保基于LLM的重构的安全性和正确性。

提出的方法

• 从20个开源Java项目中构建了一个高质量的180个真实重构的语料库，由人类专家手动验证。
• 使用GPT-4和Gemini作为LLMs，分析完整的Java源文件，识别重构机会和解决方案。
• 采用提示工程——特别是指定重构子类别并缩小搜索空间——以提升LLM性能。
• 提出RefactoringMirror，一种检测并重新应用的策略，通过既定且经过测试的重构引擎重新应用LLM建议的重构。
• 对LLMs识别机会的成功率和推荐解决方案的质量进行了定量评估。
• 通过三位人类专家进行人工评估，以判断LLM生成的重构解决方案的正确性和质量。

实验结果

研究问题

• RQ1 RQ1：在真实世界的Java代码中，LLMs（GPT-4和Gemini）在识别重构机会方面的表现如何？
• RQ2 RQ2：与人类专家相比，GPT-4和Gemini推荐高质量重构解决方案的表现如何？
• RQ3 RQ3：基于LLM的重构存在哪些风险，特别是在引入功能错误或语法错误方面？
• RQ4 RQ4：检测并重新应用的策略是否能有效缓解基于LLM的重构中的幻觉问题，同时保留LLM智能的优势？

主要发现

• 通过优化提示工程，ChatGPT识别出了180个重构机会中的86.7%，相较于初始15.6%的成功率有显著提升。
• Gemini仅识别出180个重构机会中的7个，表明在未进行提示优化的情况下性能有限。
• ChatGPT为180次重构推荐了176个解决方案，其中63.6%与人类生成的解决方案相当或更优。
• ChatGPT生成的176个解决方案中有13个不安全——要么引入语法错误，要么改变了功能；Gemini生成的137个解决方案中有9个不安全。
• 所提出的RefactoringMirror策略成功重新应用了94.3%的LLM建议的重构，并消除了所有不安全更改。
• 本研究证明，当结合通过既定重构引擎进行的安全检查时，LLMs可作为自动化重构中可行且高潜力的辅助工具。