[Paper Review] Fixing Rust Compilation Errors using LLMs

Pantazis Deligiannis, Akash Lal+2 more|arXiv (Cornell University)|Aug 9, 2023

Software Engineering Research4 citations

TL;DR

This paper presents RustAssistant, an LLM-based tool that automatically fixes Rust compilation errors using advanced prompting and iterative LLM refinement. It achieves up to 74% accuracy on real-world errors from popular open-source repositories, demonstrating the effectiveness of LLMs in debugging complex systems programming issues with strong safety guarantees.

ABSTRACT

The Rust programming language, with its safety guarantees, has established itself as a viable choice for low-level systems programming language over the traditional, unsafe alternatives like C/C++. These guarantees come from a strong ownership-based type system, as well as primitive support for features like closures, pattern matching, etc., that make the code more concise and amenable to reasoning. These unique Rust features also pose a steep learning curve for programmers. This paper presents a tool called RustAssistant that leverages the emergent capabilities of Large Language Models (LLMs) to automatically suggest fixes for Rust compilation errors. RustAssistant uses a careful combination of prompting techniques as well as iteration with an LLM to deliver high accuracy of fixes. RustAssistant is able to achieve an impressive peak accuracy of roughly 74% on real-world compilation errors in popular open-source Rust repositories. We plan to release our dataset of Rust compilation errors to enable further research.

Motivation & Objective

• To address the steep learning curve of Rust's ownership and borrowing system, which often leads to confusing compilation errors for beginners.
• To reduce the time and cognitive load required to debug Rust programs by automating the suggestion of fixes for compilation errors.
• To evaluate the effectiveness of LLMs in understanding and resolving complex Rust type system and borrow checker errors.
• To build and release a dataset of real-world Rust compilation errors for future research in program repair and LLM-assisted debugging.

Proposed method

• RustAssistant uses a custom changelog-based prompt format to guide the LLM in understanding and modifying code incrementally.
• It applies iterative prompting, where the LLM generates fixes and the system validates and refines them through multiple rounds.
• The tool integrates with the Rust compiler to detect compilation errors and then uses the LLM to suggest and apply patches.
• It leverages GPT-3.5 and GPT-4 models, with prompt engineering tailored to Rust's unique type system and error semantics.
• The system evaluates fixes by recompiling the modified code to verify correctness.
• A dataset of 182 GitHub commits with real compilation errors is collected from top crates on crates.io to evaluate the approach.

Experimental results

Research questions

• RQ1 Can LLMs effectively understand and fix complex Rust compilation errors, particularly those involving ownership and borrowing?
• RQ2 How does prompt engineering and iterative refinement impact the accuracy of LLM-generated fixes for Rust errors?
• RQ3 What is the performance difference between GPT-3.5 and GPT-4 in fixing real-world Rust compilation errors?
• RQ4 To what extent can LLM-based tools generalize beyond compilation errors to other types of Rust development issues?

Key findings

• RustAssistant achieved a peak accuracy of 73.63% on GitHub commits containing real-world compilation errors from top Rust crates.
• On Stack Overflow programs, the tool fixed 72% of the compilation errors when using GPT-4.
• For micro-benchmarks covering 270 of 506 official Rust error codes, GPT-4 achieved a 92.59% fix rate.
• GPT-4 significantly outperformed GPT-3.5 in all evaluation settings, confirming the impact of model scale and reasoning capability.
• Ablation studies showed that iterative prompting and custom prompt engineering were critical for high fix accuracy.
• The authors released a dataset of 182 real-world compilation errors from popular Rust repositories to support future research.