[Paper Review] GitHub Copilot: the perfect Code compLeeter?

Ilja Siroš, Dave Singelée+1 more|arXiv (Cornell University)|Jun 17, 2024

Distributed and Parallel Computing Systems4 citations

TL;DR

This paper evaluates GitHub Copilot's code generation quality across 1,760 LeetCode problems in Java, C++, Python3, and Rust using an automated framework. It finds Copilot is most reliable in Java and C++, generates more efficient code than average human submissions, but struggles with correctness and consistency in Python3 and Rust, and its top-ranked suggestions are not always the best.

ABSTRACT

This paper aims to evaluate GitHub Copilot's generated code quality based on the LeetCode problem set using a custom automated framework. We evaluate the results of Copilot for 4 programming languages: Java, C++, Python3 and Rust. We aim to evaluate Copilot's reliability in the code generation stage, the correctness of the generated code and its dependency on the programming language, problem's difficulty level and problem's topic. In addition to that, we evaluate code's time and memory efficiency and compare it to the average human results. In total, we generate solutions for 1760 problems for each programming language and evaluate all the Copilot's suggestions for each problem, resulting in over 50000 submissions to LeetCode spread over a 2-month period. We found that Copilot successfully solved most of the problems. However, Copilot was rather more successful in generating code in Java and C++ than in Python3 and Rust. Moreover, in case of Python3 Copilot proved to be rather unreliable in the code generation phase. We also discovered that Copilot's top-ranked suggestions are not always the best. In addition, we analysed how the topic of the problem impacts the correctness rate. Finally, based on statistics information from LeetCode, we can conclude that Copilot generates more efficient code than an average human.

Motivation & Objective

• To systematically evaluate GitHub Copilot’s reliability, correctness, and efficiency in generating code for programming problems.
• To investigate how code quality varies across programming languages, problem difficulty, topic, and suggestion rank.
• To compare Copilot-generated code performance against average human submissions on time and memory efficiency.
• To assess the impact of prompt context and potential recitation (repetition of training data) in Copilot outputs.
• To provide a reproducible, automated benchmark for AI-generated code evaluation using LeetCode as a testbed.

Proposed method

• Automated framework to generate and test Copilot suggestions for all publicly available, single-function LeetCode problems in four languages.
• For each problem, Copilot was queried multiple times to collect all suggested solutions, which were then submitted to LeetCode’s automated test suite.
• Correctness was determined by LeetCode’s test results; efficiency was measured via runtime and memory consumption metrics provided by LeetCode.
• Statistical analysis compared Copilot’s performance across programming languages, problem difficulty levels (easy/medium/hard), and problem topics.
• Recitation detection involved searching for LeetCode links in Copilot outputs to identify potential data leakage from training data.
• Results were aggregated across 1760 problems per language, totaling over 50,000 submissions over a two-month period.

Figure 1: Example of a LeetCode problem. On the left part of the figure, the problem description is shown and some examples are mentioned together with the constraints imposed on the solution by LeetCode. The right part of the figure shows the solution template for the chosen programming language, c

Experimental results

Research questions

• RQ1 RQ1: How reliable is Copilot in the code generation phase, and how does the number of suggestions and failure rate vary by programming language?
• RQ2 RQ2: How does the correctness of Copilot’s suggestions depend on the programming language, problem difficulty, suggestion rank, and problem topic?
• RQ3 RQ3: How does the time and memory efficiency of Copilot’s correct solutions compare to the average human submission on LeetCode?
• RQ4 RQ4: To what extent does recitation (repetition of known solutions) occur in Copilot’s outputs, especially for LeetCode problems?

Key findings

• Copilot demonstrated higher reliability in code generation for Java and C++ compared to Python3 and Rust, with fewer generation failures and more suggestions per problem.
• Across all languages, Copilot achieved a correctness rate exceeding 72%, with significantly higher success in Java and C++ than in Python3 and Rust.
• The top-ranked Copilot suggestion was not always the best; multiple suggestions needed to be evaluated to find the optimal solution.
• Among problem topics, Bucket Sort had the highest correctness rate, while Tree-related problems had the lowest.
• Copilot-generated code was more time- and memory-efficient than the average human submission on LeetCode, indicating superior performance in efficiency metrics.
• Recitation was observed in 5 out of 1,760 problems, all in Python3, where Copilot included non-working LeetCode links in its output, suggesting possible exposure to training data.

Figure 2: The code of the problem before invoking Copilot. The problem’s description is added as a comment above the code template to give Copilot a context to generate a solution. Before invoking Copilot, the mouse cursor is placed inside the function that we want Copilot to generate. In this examp