[Paper Review] Can ChatGPT Pass An Introductory Level Functional Language Programming Course?

Chuqin Geng, Yihan Zhang+2 more|arXiv (Cornell University)|Apr 29, 2023

Artificial Intelligence in Healthcare and Education15 citations

TL;DR

The paper evaluates ChatGPT as a student in an OCaml-based introductory functional programming course, finding an overall grade of 67% (B−) and a class rank improvement from 220nd to 155th with prompt-engineering assisted prompts.

ABSTRACT

The recent introduction of ChatGPT has drawn significant attention from both industry and academia due to its impressive capabilities in solving a diverse range of tasks, including language translation, text summarization, and computer programming. Its capability for writing, modifying, and even correcting code together with its ease of use and access is already dramatically impacting computer science education. This paper aims to explore how well ChatGPT can perform in an introductory-level functional language programming course. In our systematic evaluation, we treated ChatGPT as one of our students and demonstrated that it can achieve a grade B- and its rank in the class is 155 out of 314 students overall. Our comprehensive evaluation provides valuable insights into ChatGPT's impact from both student and instructor perspectives. Additionally, we identify several potential benefits that ChatGPT can offer to both groups. Overall, we believe that this study significantly clarifies and advances our understanding of ChatGPT's capabilities and potential impact on computer science education.

Motivation & Objective

• Assess ChatGPT's ability to solve real course problems in an OCaml-based introductory functional programming course.
• Compare ChatGPT's performance to actual student outcomes from Fall 2022 course data.
• Explore the effects of prompt engineering techniques on ChatGPT's problem-solving quality.
• Investigate which topics ChatGPT handles well and where it struggles (e.g., type inference).
• Evaluate potential benefits and limitations of using ChatGPT as a learning or instructional aid.

Proposed method

• Treat ChatGPT as a course student and have it complete all ten homework assignments and relevant exam questions from Fall 2022 (OCaml-based course).
• Use unassisted vs. assisted (prompt-engineered) modes to evaluate performance.
• Classify errors into syntax, compilation, and logical categories to analyze failures.
• Apply four prompt-engineering techniques (paraphrasing, natural-language hints, teaching by example, and providing test cases) to assess performance gains.
• Grade ChatGPT's solutions using the same rubric as human students and compare rankings.
• Analyze exam components (conceptual, code completion, and induction proof) to gauge understanding beyond code generation.

Experimental results

Research questions

• RQ1 Can ChatGPT achieve comparable grades to real students in an introductory functional programming course?
• RQ2 How does ChatGPT perform on assignments and exams requiring OCaml knowledge, recursion, higher-order functions, and type systems?
• RQ3 To what extent can prompt engineering improve ChatGPT's problem-solving quality and course ranking?
• RQ4 What are ChatGPT's strengths (e.g., coding tasks) and weaknesses (e.g., type inference) in this educational setting?

Key findings

• ChatGPT attains an overall course grade of 67% (B−) in unassisted mode and ranks 155th out of 314 students after assisted prompting.
• ChatGPT passes 16 of 31 homework problems in unassisted mode, with higher performance on higher-order functions, tail recursion, lazy programming, and continuations, but struggles with type inference and larger tasks like interpreters.
• Assisted prompting (paraphrasing, hints, examples, and test cases) improves exam and homework performance, boosting final exam score from 64.4 to 81.0 (uncompared total) and midterm from 74.5 to 79.5 in the two experiments.
• ChatGPT's final exam performance in the assisted mode yields a total score of 81.0 vs 64.4 in unassisted, with notable gains in conceptual, code completion, and induction-proof components.
• Ranking improves under assisted mode (final exam: 170th unassisted vs 121st assisted; total: 220th unassisted vs 155th assisted).
• Overall, ChatGPT demonstrates substantial capability in generating correct and elegant solutions for many OCaml problems, while type-checking and large-scale tasks remain challenging.