[Paper Review] A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functions

Giuseppe Destefanis, Silvia Bartolucci+1 more|arXiv (Cornell University)|May 16, 2023

Scientific Computing and Data Management17 citations

TL;DR

The paper compares GPT-3.5 and Bard in generating Java code from CodingBat function descriptions; GPT-3.5 achieves about 90.6% correctness, while Bard achieves 53.1%.

ABSTRACT

This paper evaluates the capability of two state-of-the-art artificial intelligence (AI) models, GPT-3.5 and Bard, in generating Java code given a function description. We sourced the descriptions from CodingBat.com, a popular online platform that provides practice problems to learn programming. We compared the Java code generated by both models based on correctness, verified through the platform's own test cases. The results indicate clear differences in the capabilities of the two models. GPT-3.5 demonstrated superior performance, generating correct code for approximately 90.6% of the function descriptions, whereas Bard produced correct code for 53.1% of the functions. While both models exhibited strengths and weaknesses, these findings suggest potential avenues for the development and refinement of more advanced AI-assisted code generation tools. The study underlines the potential of AI in automating and supporting aspects of software development, although further research is required to fully realize this potential.

Motivation & Objective

• Evaluate GPT-3.5 and Bard capabilities to generate Java code from function descriptions sourced from CodingBat.com.
• Assess correctness of generated code using CodingBat.com's real-time test system.
• Identify categories where each model excels or struggles to inform AI-assisted code generation.

Proposed method

• Collect 64 Java function descriptions from five CodingBat sections (Warmup, String-3, Array-3, Functional-2, Recursion-2).
• Prompt GPT-3.5 and Bard with each description to generate Java code.
• Evaluate correctness of generated code using CodingBat.com test cases.
• Apply McNemar test to compare model success rates on the same function descriptions.

Experimental results

Research questions

• RQ1 How do GPT-3.5 and Bard compare in producing correct Java code from function descriptions?
• RQ2 Does GPT-3.5 consistently outperform Bard across problem categories, and are there statistically significant differences?
• RQ3 Which problem categories pose the most difficulty for both models?

Key findings

• GPT-3.5 generated correct code for about 90.6% of the 64 descriptions; Bard for 53.1%.
• GPT-3.5 outperformed Bard in four of five problem categories.
• Bard struggled in more complex categories (notably String-3, Array-3, and Recursion-2) and produced correct code in Functional-2 for all descriptions.
• Both models correctly code for all Functional-2 tasks; not all problems were solved correctly by either model in other categories.
• McNemar test indicates a statistically significant difference in success rates, with GPT-3.5 outperforming Bard (p = 0.0001768).
• Unsuccessful examples illustrate Bard’s incorrect solution versus GPT-3.5’s correct approach (e.g., front3).