[Paper Review] Comparing large language models and human programmers for generating programming code

Wenpin Hou, Zhicheng Ji|arXiv (Cornell University)|Mar 1, 2024

Software Engineering Research4 citations

TL;DR

This study systematically evaluates seven large language models (LLMs) in generating programming code using diverse prompt strategies, programming languages, and task difficulties. GPT-4, especially with an optimized prompt strategy, outperforms 85% of human participants in LeetCode and GeeksforGeeks coding contests, demonstrates strong cross-language code translation, learns from errors, and generates computationally efficient code, positioning it as a highly capable assistant for software development.

ABSTRACT

We systematically evaluated the performance of seven large language models in generating programming code using various prompt strategies, programming languages, and task difficulties. GPT-4 substantially outperforms other large language models, including Gemini Ultra and Claude 2. The coding performance of GPT-4 varies considerably with different prompt strategies. In most LeetCode and GeeksforGeeks coding contests evaluated in this study, GPT-4 employing the optimal prompt strategy outperforms 85 percent of human participants. Additionally, GPT-4 demonstrates strong capabilities in translating code between different programming languages and in learning from past errors. The computational efficiency of the code generated by GPT-4 is comparable to that of human programmers. These results suggest that GPT-4 has the potential to serve as a reliable assistant in programming code generation and software development.

Motivation & Objective

• To systematically evaluate the performance of seven LLMs in generating programming code across multiple programming languages and task difficulties.
• To investigate how different prompt strategies affect LLM coding performance, especially compared to standard prompting.
• To directly compare LLM-generated code performance with that of human programmers in real-world coding contests.
• To assess LLMs' ability to translate code across programming languages and their computational efficiency relative to human-written code.
• To examine whether LLMs can learn from past errors and improve over successive attempts.

Proposed method

• The study evaluated LLMs using 100 programming tasks from 25 randomly selected LeetCode contests and a smaller set from GeeksforGeeks, all post-September 2021 to align with GPT-4’s knowledge cutoff.
• Five distinct prompt strategies were developed and tested: standard prompt, feedback with chain-of-thought (feedback CI), feedback with chain-of-thought and self-consistency, and two variants of few-shot prompting.
• LLM performance was measured via one-attempt and five-attempt success rates, salvage rates (success on later attempts), and computational efficiency (execution time and memory usage percentile).
• For code translation, GPT-4 generated Python3 code using the feedback CI prompt and then translated it to Java, JavaScript, and C++, evaluating success rates by task difficulty.
• Computational efficiency was assessed by comparing GPT-4-generated code (before and after self-optimization) to human-written code on the same tasks, using percentile rankings for runtime and memory usage.
• Human performance was benchmarked using official contest rankings from LeetCode and GeeksforGeeks, with LLM rankings determined by percentile comparison to human participants.

Figure 1: A schematic illustrating the prompt strategies that were developed and assessed in this study.

Experimental results

Research questions

• RQ1 How does the choice of prompt strategy affect the coding performance of large language models across different task difficulties?
• RQ2 To what extent does GPT-4 outperform human programmers in real coding contests from LeetCode and GeeksforGeeks when using optimal prompting?
• RQ3 Can large language models effectively translate code between different programming languages, and how does translation success vary with task difficulty?
• RQ4 How does the computational efficiency of code generated by LLMs compare to that of human-written code, both before and after self-optimization?
• RQ5 Can LLMs learn from and improve on previous errors, as measured by salvage rates across multiple attempts?

Key findings

• GPT-4, when using the optimal prompt strategy (feedback with chain-of-thought and self-consistency), achieved a one-attempt success rate of 86.2% on LeetCode’s easy tasks, 60.0% on medium, and 14.3% on hard tasks—surpassing 85% of human participants in most contests.
• The salvage rate for GPT-4 increased significantly with the number of attempts, reaching over 90% on medium and hard tasks by the fifth attempt, indicating strong error recovery and learning capability.
• GPT-4 demonstrated high code translation success: 80% of correctly generated Python3 code was successfully translated to other languages (Java, JavaScript, C++) for medium-difficulty tasks, with strong performance even when the original code was incorrect.
• The computational efficiency of GPT-4-generated code was comparable to human code—after self-optimization, GPT-4’s code ranked in the 90th percentile or higher for both runtime and memory usage across all task difficulties.
• GPT-4 outperformed other LLMs, including Gemini Ultra and Claude 2, across all metrics, with the largest performance gap observed in complex, hard-difficulty tasks.
• The study confirmed that prompt engineering significantly enhances LLM coding performance, with the feedback CI strategy yielding the highest success and salvage rates across all models.

Figure 2: Five-attempt and one-attempt success rates for tasks with varying difficulty levels. Each row represents a different prompt strategy and a different LLM. The text within each bar displays the success rate, the number of programming tasks solved, and the total number of tasks. Rows are orde