[Paper Review] Exploring the Verifiability of Code Generated by GitHub Copilot

Dakota Wong, Austin Kothig+1 more|arXiv (Cornell University)|Sep 5, 2022

Software Engineering Research4 citations

TL;DR

This study evaluates the verifiability of code generated by GitHub Copilot by formally verifying six implementations using Dafny against hand-crafted specifications. It found that 4 out of 6 Copilot-generated solutions were successfully verified, indicating that while Copilot produces usable code, it requires expert-level verification and should not be trusted without rigorous validation.

ABSTRACT

GitHub's Copilot generates code quickly. We investigate whether it generates good code. Our approach is to identify a set of problems, ask Copilot to generate solutions, and attempt to formally verify these solutions with Dafny. Our formal verification is with respect to hand-crafted specifications. We have carried out this process on 6 problems and succeeded in formally verifying 4 of the created solutions. We found evidence which corroborates the current consensus in the literature: Copilot is a powerful tool; however, it should not be "flying the plane" by itself.

Motivation & Objective

• To assess the formal verifiability of code generated by GitHub Copilot using automated verification tools.
• To determine whether Copilot-generated code meets correctness requirements through formal specification and proof.
• To evaluate the practicality of integrating Copilot-generated code into production systems without verification.
• To explore the challenges developers face when verifying AI-generated code, especially in recursive or complex algorithms.
• To understand the limitations of Copilot in generating code that is both correct and amenable to formal verification.

Proposed method

• Selected six algorithmic problems from LeetCode, including binary search and max-heapify, as test cases.
• Used GitHub Copilot to generate Python implementations based on natural language prompts.
• Hand-crafted formal specifications in Dafny to describe the intended behavior of each algorithm.
• Applied Dafny’s verification framework to check whether Copilot’s implementations satisfy the specifications.
• Used a Python-mediated translation to avoid Dafny’s complexity with pointer arithmetic and memory management.
• Focused on identifying invariants and loop structures to guide the verification process, especially for recursive implementations.

Experimental results

Research questions

• RQ1 Can Dafny formally verify GitHub Copilot-generated code against hand-crafted specifications?
• RQ2 How many of the Copilot-generated implementations were successfully verified, and what factors influenced verification success?
• RQ3 What are the primary challenges in verifying AI-generated code, particularly recursive or non-trivial algorithms?
• RQ4 How does the quality of Copilot-generated code compare to human-written code in terms of verifiability and correctness?
• RQ5 To what extent does Copilot’s training data influence its ability to generate verifiable code for well-known algorithmic problems?

Key findings

• Four out of six Copilot-generated implementations were successfully formally verified using Dafny, demonstrating that some AI-generated code is amenable to formal verification.
• The verification of recursive implementations, such as the max-heapify function, proved particularly challenging and was not completed, despite the code being potentially correct.
• Copilot generated sub-optimal and sometimes incorrect solutions, such as in the max-heapify case, indicating that not all generated code is suitable for direct use.
• The lack of support for square root in Dafny introduced a tool-specific limitation, though this did not prevent verification in most cases.
• Verification success depended heavily on the clarity of invariants and the structure of the code, with iterative solutions being easier to verify than recursive ones.
• The study suggests that while Copilot accelerates prototyping, it should not be used without rigorous vetting, including formal verification or extensive testing.