[Paper Review] Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?

Jaromír Šavelka, Arav Agarwal+3 more|arXiv (Cornell University)|Mar 16, 2023

Teaching and Learning Programming33 references7 citations

TL;DR

GPT models (text-davinci-003) cannot pass the full range of Python course assessments, but can acquire a non-trivial portion of the overall score, indicating potential as a learning assistant rather than a complete solver.

ABSTRACT

We evaluated the capability of generative pre-trained transformers (GPT), to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. Discussions of potential uses (e.g., exercise generation, code explanation) and misuses (e.g., cheating) of this emerging technology in programming education have intensified, but to date there has not been a rigorous analysis of the models' capabilities in the realistic context of a full-fledged programming course with diverse set of assessment instruments. We evaluated GPT on three Python courses that employ assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Further, we studied if and how successfully GPT models leverage feedback provided by an auto-grader. We found that the current models are not capable of passing the full spectrum of assessments typically involved in a Python programming course (<70% on even entry-level modules). Yet, it is clear that a straightforward application of these easily accessible models could enable a learner to obtain a non-trivial portion of the overall available score (>55%) in introductory and intermediate courses alike. While the models exhibit remarkable capabilities, including correcting solutions based on auto-grader's feedback, some limitations exist (e.g., poor handling of exercises requiring complex chains of reasoning steps). These findings can be leveraged by instructors wishing to adapt their assessments so that GPT becomes a valuable assistant for a learner as opposed to an end-to-end solution.

Motivation & Objective

• Evaluate GPT capabilities on a full spectrum of Python course assessments (MCQs and coding tasks) used in real postsecondary courses.
• Assess GPT’s ability to leverage auto-grader feedback to improve solutions.
• Identify limitations and implications for instructional design and assessment security in programming education.

Proposed method

• Use text-davinci-003 (GPT-3.5 family) with MCQ prompts (temperature 0.0, max_tokens 500) and coding task prompts (max_tokens up to 2000, temperature 0.7).
• Apply the same assessment regimen as human learners, including auto-grader feedback and iterative submissions.
• Dataset comprises 599 exercises (MCQs and coding tasks) from three Python courses: PE1, PE2, and PPP.
• Compare GPT performance to earlier model (text-davinci-002) to measure progress.
• Analyze whether GPT can pass course-level assessments and how feedback influences corrections.

Experimental results

Research questions

• RQ1 RQ1: How reliably can GPT generate correct answers to MCQ assessments used in introductory and intermediate Python courses?
• RQ2 RQ2: How reliably can GPT produce solutions to complex Python coding tasks from human-like instructions?
• RQ3 RQ3: How well can GPT utilize auto-grader feedback to improve defective solutions?

Key findings

• GPT-3.5 (text-davinci-003) answered 341 of 530 MCQs (64.3%), improving over the 301/530 (56.8%) by the earlier model (td-002).
• In PE1, td-003 passed only the first module and averaged 56.1% overall, failing subsequent module tests.
• In PE2, td-003 passed 3 of 4 module tests and averaged 67.9% overall (65.0% on the Summary Test).
• In PPP, GPT passed 4 of 8 course units and achieved an overall 67.3% (course total) under the stated grading scheme, with project and reflection components weighted at 0.8 and 0.2 respectively.
• GPT demonstrated the ability to improve via auto-grader feedback (e.g., project scores rising from 53.6% on first submissions to 66.4% after revisions); however, it struggles with tasks requiring long chains of reasoning or nuanced output understanding.
• Limitations include difficulty with exercises needing complex reasoning steps, multi-artifact integration, or requiring nontrivial output validation.