[Paper Review] Computing Education in the Era of Generative AI

Paul Denny, James Prather+8 more|arXiv (Cornell University)|Jun 5, 2023

Software Engineering Research10 citations

TL;DR

This paper surveys challenges and opportunities that generative AI and code-generation models (e.g., Codex, Copilot, GPT-4) pose for introductory programming education, and explores how these models can generate learning resources and improve feedback while raising integrity and licensing concerns.

ABSTRACT

The computing education community has a rich history of pedagogical innovation designed to support students in introductory courses, and to support teachers in facilitating student learning. Very recent advances in artificial intelligence have resulted in code generation models that can produce source code from natural language problem descriptions -- with impressive accuracy in many cases. The wide availability of these models and their ease of use has raised concerns about potential impacts on many aspects of society, including the future of computing education. In this paper, we discuss the challenges and opportunities such models present to computing educators, with a focus on introductory programming classrooms. We summarize the results of two recent articles, the first evaluating the performance of code generation models on typical introductory-level programming problems, and the second exploring the quality and novelty of learning resources generated by these models. We consider likely impacts of such models upon pedagogical practice in the context of the most recent advances at the time of writing.

Motivation & Objective

• Assess how large language models perform on typical CS1 programming problems and exams.
• Investigate academic integrity, licensing, and potential student over-reliance risks from AI-generated code.
• Explore opportunities to use AI to generate learning resources, explanations, and improved programming error messages.
• Discuss pedagogy adjustments and future directions for introductory computing education in light of AI advances.

Proposed method

• Review of two recent studies evaluating code-generation models on CS1 problems and resources.
• Experimental replication: Codex evaluated on Python CS1 exam questions against student performance.
• Analysis of multiple Rainfall problem variants to study solution diversity and execution against test cases.
• Assessment of learning resource generation (programming exercises and code explanations) and error-message improvements.
• Discussion of academic integrity, licensing, and bias considerations relevant to AI-generated code.

Experimental results

Research questions

• RQ1 How well do code-generation models perform on typical CS1 programming problems compared with novice programmers?
• RQ2 What are the implications of AI-generated code for academic integrity, plagiarism detection, and licensing in introductory computing courses?
• RQ3 How can LLM-based systems generate effective learning resources (exercises, explanations) for CS1, and how reliable are they?
• RQ4 What pedagogical adjustments could leverage AI while mitigating risks such as learner over-reliance and insecure code?
• RQ5 What are the potential biases and security concerns associated with AI-generated code in education?

Key findings

• Codex achieved 78.5% (15.7/20) on Exam 1 and 78.0% (19.5/25) on Exam 2, ranking 17th among 71 students in the CS1 course analyzed.
• Codex's performance varied with prompt constraints and sometimes failed when language features were restricted or when output required specific formatting (ASCII art).
• Across 50 Rainfall problem variants (350 evaluations), Codex averaged about 50% and produced diverse solutions; 84.6% of generated exercises included a sample solution and 89.7% of those solutions were executable.
• In generated explanations, 90% of line-by-line explanations covered all code parts and about 70% were correct per line; higher-quality explanations emerged with newer models (e.g., ChatGPT) compared to earlier Codex variants.
• AI-assisted resource generation showed potential to create novel, thematically aligned exercises and tests with substantial coverage of concepts, enabling broader pedagogical experimentation.
• Risks identified include academic misconduct concerns, license and attribution issues for generated code, deployment of insecure code by novices, and bias in AI outputs; calls for careful policy and oversight.