[Paper Review] Comparing Software Developers with ChatGPT: An Empirical Investigation

Nathália Nascimento, Paulo Alencar+1 more|arXiv (Cornell University)|May 19, 2023

Artificial Intelligence in Healthcare and Education32 citations

TL;DR

This paper empirically compares ChatGPT-generated code with code from experienced and novice programmers on LeetCode, focusing on performance and memory efficiency to assess human-AI teamwork in coding tasks.

ABSTRACT

The advent of automation in particular Software Engineering (SE) tasks has transitioned from theory to reality. Numerous scholarly articles have documented the successful application of Artificial Intelligence to address issues in areas such as project management, modeling, testing, and development. A recent innovation is the introduction of ChatGPT, an ML-infused chatbot, touted as a resource proficient in generating programming codes and formulating software testing strategies for developers and testers respectively. Although there is speculation that AI-based computation can increase productivity and even substitute software engineers in software development, there is currently a lack of empirical evidence to verify this. Moreover, despite the primary focus on enhancing the accuracy of AI systems, non-functional requirements including energy efficiency, vulnerability, fairness (i.e., human bias), and safety frequently receive insufficient attention. This paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration, enhancing the reliability of AI-based methods, and understanding task suitability for humans or AI. Furthermore, it facilitates the effective implementation of cooperative work structures and human-in-the-loop processes. This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics. The empirical study includes a case of assessing ChatGPT-generated code versus code produced by developers and uploaded in Leetcode.

Motivation & Objective

• Motivate the need to evaluate AI-based coding tools alongside human developers across non-functional criteria.
• Investigate whether ChatGPT can outperform human programmers in coding problems of varying difficulty.
• Assess performance, memory efficiency, and potential task allocation for human-in-the-loop SE tasks.

Proposed method

• Use ChatGPT-4 to generate C++ solutions for a recent LeetCode contest with easy, medium, and hard problems.
• Compare ChatGPT solutions to solutions from experienced and novice contest programmers on LeetCode metrics.
• Apply statistical tests (t-statistics, 0.01 significance) to evaluate performance and memory efficiency differences.
• Analyze time and space complexity of generated solutions and observe variability across problems.
• Discuss threats to validity including dataset provenance and potential reuse of prior solutions.

Experimental results

Research questions

• RQ1 RQ1: Does ChatGPT improve performance (runtime) compared to experienced contest programmers on easy/medium/hard problems?
• RQ2 RQ2: Does ChatGPT improve memory efficiency compared to experienced contest programmers on easy/medium/hard problems?
• RQ3 RQ3: Does ChatGPT improve performance compared to novice contest programmers on easy/medium/hard problems?
• RQ4 RQ4: Does ChatGPT improve memory efficiency compared to novice contest programmers on easy/medium/hard problems.

Key findings

• ChatGPT solutions beat novice programmers in performance on easy and medium problems.
• ChatGPT solutions show better memory efficiency than novices on a medium problem.
• There is no strong evidence that ChatGPT outperforms experienced contest programmers in performance.
• For hard problems, ChatGPT did not reliably outperform experienced programmers and some problems remained unsolved by all groups.
• Across problems, ChatGPT achieved substantial relative standing against prior solutions in runtime and memory, with some variability due to problem characteristics and platform evaluation.