[Paper Review] The Robots are Here: Navigating the Generative AI Revolution in Computing Education

James Prather, Paul Denny+13 more|arXiv (Cornell University)|Oct 1, 2023

Teaching and Learning ProgrammingComputer Science150 references8 citations

TL;DR

A comprehensive working group report synthesizing literature, stakeholder attitudes, pedagogy shifts, ethics, replication, and benchmark findings on large language models (LLMs) in computing education. It reviews 71 primary articles, surveys 20 countries, and includes educator interviews, ethical framing, and performance benchmarking.

ABSTRACT

Recent advancements in artificial intelligence (AI) are fundamentally reshaping computing, with large language models (LLMs) now effectively being able to generate and interpret source code and natural language instructions. These emergent capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of LLMs in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents who have already adapted their curricula and assessments. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of LLMs on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving. Our aim is that this report will serve as a focal point for both researchers and practitioners who are exploring, adapting, using, and evaluating LLMs and LLM-based tools in computing classrooms.

Motivation & Objective

• Summarize the state of research on LLMs in computing education up to August 2023.
• Capture international attitudes of students and instructors toward LLMs in education.
• Identify instructional approaches and curricular adaptations influenced by LLMs.
• Frame ethical and policy considerations using the ACM Code of Ethics.
• Benchmark LLM performance on computing education datasets and discuss implications for practice.

Proposed method

• Conduct a scoping literature review of papers on LLMs in computing education up to August 2023.
• Perform forward and backward snowballing to identify a comprehensive set of relevant works.
• Classify literature into categories (performance assessment, teaching materials, student work analysis, programmer-LLM interactions, and surveys).
• Incorporate an international survey of students and instructors from 20 countries.
• Conduct in-depth interviews with 22 computing educators across five continents.
• Frame ethical discussion via ACM Code of Ethics and provide policy/practice recommendations.
• Replicate prior work using new LLMs to assess reproducibility and describe datasets.

Figure 1 . Phases of the literature review.

Experimental results

Research questions

• RQ1 What is the current state and scope of research on LLMs in computing education as of mid-2023?
• RQ2 How do students and instructors across multiple countries perceive and adopt LLMs in computing education?
• RQ3 What instructional practices and curricular adaptations are emerging due to LLM capabilities?
• RQ4 What ethical considerations arise from LLM use in computing education, and how are institutions addressing them?
• RQ5 How do contemporary LLMs perform on computing education datasets, and what does this imply for future practice?

Key findings

• Evidence shows LLMs can generate code at or above the level of average students for coding tasks, but may underperform on certain MCQs and conceptual problems.
• LLMs offer potential benefits for error explanations, personalized teaching materials, and new problem types like Prompt Problems.
• Ethical and integrity concerns are prominent, and detectors for LLM-generated text are prone to false positives; policy guidance is needed.
• Educators report both advantages and challenges in adapting curricula, with a trend toward more autonomous and customized learning experiences.
• The literature body is rapidly evolving, with 71 papers identified (38 after filtering) and a strong shift toward arXiv-distributed work, highlighting fast progression in the field.

Figure 2 . Summaries of the survey responses from 171 students and 57 instructors: 1) Students’ and instructors’ perspectives were compared along likert scale responses, 2) students ranked their help seeking preferences from 1 to 6, and 3) instructors shared their beliefs about the ethical use of Ge