[Paper Review] An Empirical Study on the Potential of LLMs in Automated Software Refactoring

Bo Liu, Yanjie Jiang+4 more|arXiv (Cornell University)|Nov 7, 2024

Digital Rights Management and Security4 citations

TL;DR

This paper evaluates the potential of large language models (LLMs) like ChatGPT and Gemini in automated software refactoring, finding that with optimized prompting, LLMs can identify 86.7% of refactoring opportunities and recommend high-quality solutions comparable to human experts. To mitigate hallucinations causing syntax or functional errors, the authors propose RefactoringMirror—a detect-and-reapply tactic that re-applies LLM-suggested refactorings via verified refactoring engines, successfully avoiding all unsafe changes.

ABSTRACT

Recent advances in large language models (LLMs), make it potentially feasible to automatically refactor source code with LLMs. However, it remains unclear how well LLMs perform compared to human experts in conducting refactorings automatically and accurately. To fill this gap, in this paper, we conduct an empirical study to investigate the potential of LLMs in automated software refactoring, focusing on the identification of refactoring opportunities and the recommendation of refactoring solutions. We first construct a high-quality refactoring dataset comprising 180 real-world refactorings from 20 projects, and conduct the empirical study on the dataset. With the to-be-refactored Java documents as input, ChatGPT and Gemini identified only 28 and 7 respectively out of the 180 refactoring opportunities. However, explaining the expected refactoring subcategories and narrowing the search space in the prompts substantially increased the success rate of ChatGPT from 15.6% to 86.7%. Concerning the recommendation of refactoring solutions, ChatGPT recommended 176 refactoring solutions for the 180 refactorings, and 63.6% of the recommended solutions were comparable to (even better than) those constructed by human experts. However, 13 out of the 176 solutions suggested by ChatGPT and 9 out of the 137 solutions suggested by Gemini were unsafe in that they either changed the functionality of the source code or introduced syntax errors, which indicate the risk of LLM-based refactoring. To this end, we propose a detect-and-reapply tactic, called RefactoringMirror, to avoid such unsafe refactorings. By reapplying the identified refactorings to the original code using thoroughly tested refactoring engines, we can effectively mitigate the risks associated with LLM-based automated refactoring while still leveraging LLM's intelligence to obtain valuable refactoring recommendations.

Motivation & Objective

• To investigate whether LLMs can accurately identify real-world refactoring opportunities in production Java code.
• To evaluate the quality of refactoring solutions recommended by LLMs compared to those created by human experts.
• To address the risk of LLM-generated refactorings introducing syntax errors or changing functionality.
• To propose and validate a detect-and-reapply strategy that ensures safety and correctness in LLM-based refactoring.

Proposed method

• Constructed a high-quality dataset of 180 real-world refactorings from 20 open-source Java projects, manually verified by human experts.
• Used GPT-4 and Gemini as LLMs to analyze entire Java source files and identify refactoring opportunities and solutions.
• Employed prompt engineering—specifically, specifying refactoring subcategories and narrowing the search space—to improve LLM performance.
• Proposed RefactoringMirror, a detect-and-reapply tactic that re-applies LLM-suggested refactorings using established, tested refactoring engines.
• Conducted a quantitative evaluation of LLMs’ success rates in identifying opportunities and quality of recommended solutions.
• Performed manual assessment by three human experts to evaluate the correctness and quality of LLM-generated refactoring solutions.

Experimental results

Research questions

• RQ1 RQ1: How well do LLMs (GPT-4 and Gemini) perform in identifying refactoring opportunities in real-world Java code?
• RQ2 RQ2: How well do GPT-4 and Gemini recommend high-quality refactoring solutions compared to human experts?
• RQ3 RQ3: What are the risks of LLM-based refactoring, particularly in terms of introducing functional or syntactic errors?
• RQ4 RQ4: Can a detect-and-reapply strategy effectively mitigate hallucinations in LLM-based refactoring while preserving the benefits of LLM intelligence?

Key findings

• With optimized prompting, ChatGPT identified 86.7% of the 180 refactoring opportunities, a significant improvement from the initial 15.6% success rate.
• Gemini identified only 7 out of 180 refactoring opportunities, indicating limited performance without prompt refinement.
• ChatGPT recommended 176 solutions for 180 refactorings, with 63.6% being comparable to or better than human-generated solutions.
• 13 of the 176 solutions from ChatGPT and 9 of the 137 from Gemini were unsafe—either introducing syntax errors or altering functionality.
• The proposed RefactoringMirror tactic successfully re-applied 94.3% of LLM-suggested refactorings and eliminated all unsafe changes.
• The study demonstrates that LLMs can be a viable, high-potential aid in automated refactoring when combined with safety checks via established refactoring engines.