[論文レビュー] ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course

Eng Lieh Ouh, Benjamin Kok Siew Gan+2 more|arXiv (Cornell University)|May 23, 2023

Artificial Intelligence in Healthcare and Education被引用数 10

ひとこと要約

本論文はChatGPT（3.5および4）が80個の学部向け演習問題のJavaプログラミング解答を生成する能力を評価し、カテゴリ別の正確性、可読性、および制限を検討する。

ABSTRACT

In this study, we assess the efficacy of employing the ChatGPT language model to generate solutions for coding exercises within an undergraduate Java programming course. ChatGPT, a large-scale, deep learning-driven natural language processing model, is capable of producing programming code based on textual input. Our evaluation involves analyzing ChatGPT-generated solutions for 80 diverse programming exercises and comparing them to the correct solutions. Our findings indicate that ChatGPT accurately generates Java programming solutions, which are characterized by high readability and well-structured organization. Additionally, the model can produce alternative, memory-efficient solutions. However, as a natural language processing model, ChatGPT struggles with coding exercises containing non-textual descriptions or class files, leading to invalid solutions. In conclusion, ChatGPT holds potential as a valuable tool for students seeking to overcome programming challenges and explore alternative approaches to solving coding problems. By understanding its limitations, educators can design coding exercises that minimize the potential for misuse as a cheating aid while maintaining their validity as assessment tools.

研究の動機と目的

• Assess how effectively ChatGPT can generate solutions for Java programming exercises in an undergraduate course.
• Compare ChatGPT-generated solutions to expected outcomes across multiple exercise categories.
• Identify strengths, limitations, and educational implications to minimize misuse while preserving learning.

提案手法

• Input weekly Java programming exercise instructions into ChatGPT (versions 3.5 and 4) with explicit instruction to output Java code.
• Execute generated Java code locally and compare results to expected outputs.
• Analyze performance across exercise categories: Java Basics, Object-Oriented Concepts, Java Classes and Methods, Java Packaging, Java Exceptions and I/O.
• Note differences between text-based inputs and non-text inputs like API docs and UML diagrams.
• Discuss implications for teaching, assessment design, and asynchronous learning support.

実験結果

リサーチクエスチョン

• RQ1 How effective is ChatGPT in generating coding solutions for Java programming exercises?
• RQ2 What are the strengths and limitations of ChatGPT across different Java programming topics?
• RQ3 Can ChatGPT provide alternative, memory-efficient solutions and debugging support while preserving assessment integrity?

主な発見

• ChatGPT can generate solutions that meet exercise requirements in many Java Basics tasks.
• ChatGPT-4 outperforms ChatGPT-3.5 on more complex or diagram-based instructions.
• Performance declines when exercises rely on non-text inputs like API documentation or UML diagrams.
• ChatGPT can offer memory-efficient alternatives (e.g., using StringBuilder) but may require follow-up prompts for optimization comparisons.
• ChatGPT-generated solutions can aid asynchronous learning and debugging, but initial inaccuracy is possible and requires student evaluation.