[Paper Review] Frustrated with Code Quality Issues? LLMs can Help!

N. K. Wadhwa, Jui Pradhan+6 more|arXiv (Cornell University)|Sep 22, 2023

Software Engineering ResearchComputer Science3 citations

TL;DR

This paper proposes CORE, a two-stage LLM-based system that uses an instruction-following language model (proposer) to generate code revisions for static analysis warnings, and a second LLM (ranker) to evaluate and rank these revisions for functional correctness and quality. CORE achieves 59.2% fix rate on Python and 76.8% on Java across multiple tools and checks, reducing false positives by 25.8% and matching specialized APR tools with far less engineering effort.

ABSTRACT

As software projects progress, quality of code assumes paramount importance as it affects reliability, maintainability and security of software. For this reason, static analysis tools are used in developer workflows to flag code quality issues. However, developers need to spend extra efforts to revise their code to improve code quality based on the tool findings. In this work, we investigate the use of (instruction-following) large language models (LLMs) to assist developers in revising code to resolve code quality issues. We present a tool, CORE (short for COde REvisions), architected using a pair of LLMs organized as a duo comprised of a proposer and a ranker. Providers of static analysis tools recommend ways to mitigate the tool warnings and developers follow them to revise their code. The \emph{proposer LLM} of CORE takes the same set of recommendations and applies them to generate candidate code revisions. The candidates which pass the static quality checks are retained. However, the LLM may introduce subtle, unintended functionality changes which may go un-detected by the static analysis. The \emph{ranker LLM} evaluates the changes made by the proposer using a rubric that closely follows the acceptance criteria that a developer would enforce. CORE uses the scores assigned by the ranker LLM to rank the candidate revisions before presenting them to the developer. CORE could revise 59.2% Python files (across 52 quality checks) so that they pass scrutiny by both a tool and a human reviewer. The ranker LLM is able to reduce false positives by 25.8% in these cases. CORE produced revisions that passed the static analysis tool in 76.8% Java files (across 10 quality checks) comparable to 78.3% of a specialized program repair tool, with significantly much less engineering efforts.

Motivation & Objective

• To reduce developer burden in fixing code quality issues flagged by static analysis tools.
• To overcome limitations of existing automated program repair (APR) tools that require extensive engineering, training data, or are constrained to specific patterns.
• To leverage natural-language fix recommendations from static analysis tools as instructions for LLMs, enabling zero-shot, few-shot, or zero-shot-with-prompting code revision.
• To improve fix quality by detecting and filtering out semantically incorrect revisions that pass static checks but alter functionality.
• To enable cross-tool, cross-language, and cross-check compatibility with minimal engineering overhead.

Proposed method

• CORE uses a duo of instruction-tuned LLMs: a proposer LLM that generates candidate code revisions based on static analysis recommendations.
• The proposer LLM is prompted with the buggy code, the quality issue description, and the natural-language fix recommendation to generate revised code.
• Generated revisions are validated against the original static analysis tool to filter out those that fail to resolve the issue.
• A second LLM (ranker) evaluates each candidate revision using a rubric aligned with human acceptance criteria, scoring for correctness and functional preservation.
• The ranker uses code diffs between original and revised code to reason about semantic changes, improving detection of subtle regressions.
• Final candidate revisions are ranked by the ranker’s scores and presented to the developer, reducing manual review effort.

Experimental results

Research questions

• RQ1 Can LLMs generate code revisions that satisfy both static analysis checks and human review criteria without fine-tuning or training data?
• RQ2 How effective is a second LLM in identifying and filtering out revisions that pass static checks but introduce unintended semantic changes?
• RQ3 Can a dual-LLM system like CORE generalize across multiple programming languages (e.g., Python and Java), static analysis tools (CodeQL and SonarQube), and diverse code quality checks?
• RQ4 How does CORE’s fix rate and false positive rate compare to rule-based automated program repair tools in terms of effectiveness and engineering cost?
• RQ5 To what extent can natural-language fix recommendations from static analysis tools be directly used as instructions for LLMs in a zero-shot setting?

Key findings

• CORE achieved a 59.2% fix rate on Python code across 52 quality checks, with revisions accepted by both static analysis tools and human reviewers.
• The ranker LLM reduced false positives by 25.8% by identifying and filtering out revisions that passed static checks but introduced unintended functionality changes.
• On the Java benchmark, CORE achieved a 76.8% fix rate across 10 quality checks, comparable to a specialized rule-based APR tool (78.3%) but with significantly less engineering effort.
• The system successfully handled diverse code quality issues across two programming languages and two static analysis tools without requiring model fine-tuning or task-specific data.
• GPT-4 as a ranker outperformed GPT-3.5-Turbo in reasoning about code diffs, demonstrating the importance of model capability in semantic evaluation.
• The use of natural-language fix recommendations as prompts enabled effective zero-shot code revision, avoiding the need for large-scale training data or pattern mining.