[论文解读] A Preliminary Analysis on the Code Generation Capabilities of GPT-3.5 and Bard AI Models for Java Functions

Giuseppe Destefanis, Silvia Bartolucci+1 more|arXiv (Cornell University)|May 16, 2023

Scientific Computing and Data Management被引用 17

一句话总结

本论文比较 GPT-3.5 与 Bard 在根据 CodingBat 函数描述生成 Java 代码方面的表现；GPT-3.5 的正确率约为 90.6%，而 Bard 为 53.1%。

ABSTRACT

This paper evaluates the capability of two state-of-the-art artificial intelligence (AI) models, GPT-3.5 and Bard, in generating Java code given a function description. We sourced the descriptions from CodingBat.com, a popular online platform that provides practice problems to learn programming. We compared the Java code generated by both models based on correctness, verified through the platform's own test cases. The results indicate clear differences in the capabilities of the two models. GPT-3.5 demonstrated superior performance, generating correct code for approximately 90.6% of the function descriptions, whereas Bard produced correct code for 53.1% of the functions. While both models exhibited strengths and weaknesses, these findings suggest potential avenues for the development and refinement of more advanced AI-assisted code generation tools. The study underlines the potential of AI in automating and supporting aspects of software development, although further research is required to fully realize this potential.

研究动机与目标

• Evaluate GPT-3.5 and Bard capabilities to generate Java code from function descriptions sourced from CodingBat.com.
• Assess correctness of generated code using CodingBat.com's real-time test system.
• Identify categories where each model excels or struggles to inform AI-assisted code generation.

提出的方法

• Collect 64 Java function descriptions from five CodingBat sections (Warmup, String-3, Array-3, Functional-2, Recursion-2).
• Prompt GPT-3.5 and Bard with each description to generate Java code.
• Evaluate correctness of generated code using CodingBat.com test cases.
• Apply McNemar test to compare model success rates on the same function descriptions.

实验结果

研究问题

• RQ1 How do GPT-3.5 and Bard compare in producing correct Java code from function descriptions?
• RQ2 Does GPT-3.5 consistently outperform Bard across problem categories, and are there statistically significant differences?
• RQ3 Which problem categories pose the most difficulty for both models?

主要发现

• GPT-3.5 generated correct code for about 90.6% of the 64 descriptions; Bard for 53.1%.
• GPT-3.5 outperformed Bard in four of five problem categories.
• Bard struggled in more complex categories (notably String-3, Array-3, and Recursion-2) and produced correct code in Functional-2 for all descriptions.
• Both models correctly code for all Functional-2 tasks; not all problems were solved correctly by either model in other categories.
• McNemar test indicates a statistically significant difference in success rates, with GPT-3.5 outperforming Bard (p = 0.0001768).
• Unsuccessful examples illustrate Bard’s incorrect solution versus GPT-3.5’s correct approach (e.g., front3).