[Paper Review] Is ChatGPT the Ultimate Programming Assistant -- How far is it?

Haoye Tian, Weiqi Lu+5 more|arXiv (Cornell University)|Apr 24, 2023

Software Engineering Research108 citations

TL;DR

The paper empirically evaluates ChatGPT as a full automated programming assistant focusing on code generation, program repair, and code summarization using LeetCode and Refactory benchmarks, highlighting both capabilities and limitations.

ABSTRACT

Recently, the ChatGPT LLM has received great attention: it can be used as a bot for discussing source code, prompting it to suggest changes, provide descriptions or even generate code. Typical demonstrations generally focus on existing benchmarks, which may have been used in model training (i.e., data leakage). To assess the feasibility of using an LLM as a useful assistant bot for programmers, we must assess its realistic capabilities on unseen problems as well as its capabilities on various tasks. In this paper, we present an empirical study of ChatGPT's potential as a fully automated programming assistant, focusing on the tasks of code generation, program repair, and code summariziation. The study investigates ChatGPT's performance on common programming problems and compares it with state-of-the-art approaches on two benchmarks. Among several findings, our study shows that ChatGPT is effective in dealing with common programming problems. However, our experiments also reveal limitations in terms of its attention span: detailed descriptions will constrain the focus of ChatGPT and prevent it from leveraging its vast knowledge to solve the actual problem. Surprisingly, we have identified the ability of ChatGPT to reason the original intention of the code. We expect future work to build on this insight for dealing with the open question of the oracle problem. Our findings contribute interesting insights to the development of LLMs for programming assistance, notably by demonstrating the importance of prompt engineering, and providing a better understanding of ChatGPT's practical applications for software engineering.

Motivation & Objective

• Assess ChatGPT's ability to generate correct and efficient code for common programming problems.
• Evaluate ChatGPT's effectiveness in repairing diverse buggy code submissions.
• Determine whether ChatGPT can identify the intention of code and provide concise explanations.
• Investigate how prompt design and input description affect ChatGPT performance in software engineering tasks.

Proposed method

• Use two LeetCode-based datasets (2016-2020 and 2022) to evaluate code generation performance.
• Use the Refactory Python bug benchmark (1783 buggy programs, 2442 correct) to evaluate program repair.
• Evaluate ChatGPT's ability to explain the intention of correct and buggy code (code summarization).
• Employ five independent prompts per task to account for randomness and report TOP-5 and AVG-5 metrics.
• Mitigate data leakage concerns by analyzing whether benchmarks may have been seen during training.

Experimental results

Research questions

• RQ1 RQ-1 How well can ChatGPT generate correct and efficient code for common programming problems?
• RQ2 RQ-2 How effectively can ChatGPT repair diverse buggy code implementations for common problems?
• RQ3 RQ-3 Can ChatGPT identify and explain the intention of given code, including faulty versions?

Key findings

• ChatGPT can generate correct code across a range of problems and outperforms some prior approaches on LeetCode data.
• ChatGPT's performance decreases for new or harder problems, indicating limited generalization to unseen problems.
• Providing lengthy descriptions can reduce ChatGPT's effectiveness; prompt engineering is crucial for good results.
• ChatGPT achieves competitive repair results, with TOP-5 success around 84% but AVG-5 around 60%, and benefits from output diversity.
• ChatGPT can identify the original intention of buggy code, offering insight into the test oracle problem.
• The study emphasizes that ChatGPT should be used as a helper rather than an autonomous programmer, and highlights the importance of multiple outputs.