[Paper Review] ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks

Giriprasad Sridhara, H. G. Ranjani+1 more|arXiv (Cornell University)|May 26, 2023

Artificial Intelligence in Healthcare and Education12 citations

TL;DR

The paper evaluates ChatGPT across 15 software engineering tasks, finding credible performance on many tasks yet limitations on others, compared to human or state-of-the-art baselines.

ABSTRACT

ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot launched by OpenAI on November 30, 2022. OpenAI's GPT-3 family of large language models serve as the foundation for ChatGPT. ChatGPT is fine-tuned with both supervised and reinforcement learning techniques and has received widespread attention for its articulate responses across diverse domains of knowledge. In this study, we explore how ChatGPT can be used to help with common software engineering tasks. Many of the ubiquitous tasks covering the breadth of software engineering such as ambiguity resolution in software requirements, method name suggestion, test case prioritization, code review, log summarization can potentially be performed using ChatGPT. In this study, we explore fifteen common software engineering tasks using ChatGPT. We juxtapose and analyze ChatGPT's answers with the respective state of the art outputs (where available) and/or human expert ground truth. Our experiments suggest that for many tasks, ChatGPT does perform credibly and the response from it is detailed and often better than the human expert output or the state of the art output. However, for a few other tasks, ChatGPT in its present form provides incorrect answers and hence is not suited for such tasks.

Motivation & Objective

• Explore the usefulness of ChatGPT for common software engineering tasks spanning development, quality assurance, and maintenance.
• Compare ChatGPT outputs with human expert ground truth and state-of-the-art tools where available.
• Identify tasks where ChatGPT performs well versus tasks where it provides incorrect or suboptimal results.

Proposed method

• Interact with ChatGPT (versions from Dec 15, 2022 and Jan 9, 2023) on up to 10 samples per task.
• Use publicly available datasets and existing state-of-the-art tools or human gold sets for comparison.
• Assess accuracy as the proportion of outputs matching the ground truth or final developer output.
• Provide qualitative observations and example conversations to illustrate strengths and weaknesses.
• Analyze a diverse set of tasks including code review, log summarization, method name suggestion, and more.

Experimental results

Research questions

• RQ1 Can ChatGPT generate accurate method names and short code summaries compared to state-of-the-art tools or human experts?
• RQ2 How well does ChatGPT perform log summarization, commit message generation, and duplicate bug report detection relative to baselines?
• RQ3 Is ChatGPT reliable for merge conflict resolution, anaphora resolution, code review, type inference, and data frame driven code generation?
• RQ4 What are the limitations of ChatGPT in vulnerability detection, refactoring, and test oracle generation?

Key findings

• ChatGPT correctly suggested method names for 9 of 10 methods, often providing more informative names than the state of the art.
• ChatGPT produced better log summaries than the state of the art on all 10 logs.
• ChatGPT generated correct commit messages in 7 of 10 cases, with some extraneous content in three cases.
• ChatGPT showed strong anaphora resolution performance, correctly resolving antecedents in all 10 requirements.
• ChatGPT matched 6 of 10 ground-truth test oracles and produced reasonable explanations for assertions.
• ChatGPT identified 4 of 10 vulnerabilities in code review and struggled with some low-level C code scenarios.