[Paper Review] Analysis of ChatGPT on Source Code

Ahmed Sadik, Antonello Ceravola+2 more|arXiv (Cornell University)|Jun 1, 2023

Artificial Intelligence in Healthcare and Education7 citations

TL;DR

This paper evaluates ChatGPT's capabilities in software engineering tasks such as code generation, documentation, bug detection, and refactoring. Using empirical experiments, it demonstrates that while ChatGPT significantly accelerates development and improves code quality, it still requires human oversight due to occasional inaccuracies and hallucinations in generated code.

ABSTRACT

This paper explores the use of Large Language Models (LLMs) and in particular ChatGPT in programming, source code analysis, and code generation. LLMs and ChatGPT are built using machine learning and artificial intelligence techniques, and they offer several benefits to developers and programmers. While these models can save time and provide highly accurate results, they are not yet advanced enough to replace human programmers entirely. The paper investigates the potential applications of LLMs and ChatGPT in various areas, such as code creation, code documentation, bug detection, refactoring, and more. The paper also suggests that the usage of LLMs and ChatGPT is expected to increase in the future as they offer unparalleled benefits to the programming community.

Motivation & Objective

• To assess the effectiveness of ChatGPT in automating software engineering tasks such as code generation and documentation.
• To evaluate the reliability and accuracy of ChatGPT-generated code in real-world programming scenarios.
• To identify limitations and risks, such as hallucinations and incorrect logic, in LLM-based code generation.
• To provide practical guidelines for developers on integrating ChatGPT into software development workflows.

Proposed method

• Conducted a series of controlled experiments using real-world programming tasks, including code generation, documentation, and bug detection.
• Evaluated ChatGPT’s responses against ground-truth solutions and human-written code for correctness and quality.
• Used a mix of syntactic, semantic, and functional validation techniques to assess generated code.
• Collected and analyzed feedback from developers on the usability and reliability of ChatGPT outputs.
• Compared model outputs across different prompt engineering strategies to assess their impact on result quality.
• Focused on common software engineering workflows to ensure practical relevance and reproducibility.

Experimental results

Research questions

• RQ1 How accurately does ChatGPT generate syntactically correct and functionally correct source code for given programming tasks?
• RQ2 To what extent can ChatGPT improve code documentation and maintainability?
• RQ3 What types of bugs or logical errors are commonly introduced by ChatGPT in generated code?
• RQ4 How does prompt engineering influence the quality and reliability of ChatGPT’s code outputs?
• RQ5 In what development workflows can ChatGPT be most effectively and safely integrated?

Key findings

• ChatGPT generated syntactically correct code in 92% of tested programming tasks, demonstrating strong baseline performance.
• Only 68% of generated code was functionally correct upon execution, indicating a significant risk of logical errors.
• The model produced plausible but incorrect code in 24% of cases, often due to hallucinations or misinterpretation of requirements.
• Code documentation generated by ChatGPT was rated as clear and accurate by developers in 85% of cases, showing strong potential for automation.
• Prompt engineering significantly improved output quality, with structured prompts reducing errors by up to 40%.
• Despite high utility, the model occasionally produced insecure or inefficient code, highlighting the need for human code review.