[Paper Review] Authoring Worked Examples for Java Programming with Human-AI Collaboration

Mohammad Hassany, Peter Brusilovsky+3 more|arXiv (Cornell University)|Dec 4, 2023

Software Engineering Research4 citations

TL;DR

This paper presents a human-AI collaborative authoring system that uses large language models (e.g., ChatGPT) to automatically generate line-by-line code explanations for Java worked examples, which instructors can then edit. A user study shows that 53.93% of participants rated AI-generated explanations as better or equally complete compared to expert explanations, demonstrating the feasibility and quality of AI-assisted example authoring in programming education.

ABSTRACT

Worked examples (solutions to typical programming problems presented as a source code in a certain language and are used to explain the topics from a programming class) are among the most popular types of learning content in programming classes. Most approaches and tools for presenting these examples to students are based on line-by-line explanations of the example code. However, instructors rarely have time to provide line-by-line explanations for a large number of examples typically used in a programming class. In this paper, we explore and assess a human-AI collaboration approach to authoring worked examples for Java programming. We introduce an authoring system for creating Java worked examples that generates a starting version of code explanations and presents it to the instructor to edit if necessary. We also present a study that assesses the quality of explanations created with this approach.

Motivation & Objective

• To address the authoring bottleneck in programming education, where instructors lack time to create detailed line-by-line explanations for code examples.
• To explore whether large language models (LLMs) can generate high-quality, pedagogically useful explanations for Java code examples.
• To evaluate the quality of AI-generated explanations compared to expert-written explanations through a user study with TAs and students.
• To assess the feasibility and effectiveness of human-AI collaboration in creating interactive, explained programming examples for educational use.
• To inform future design of authoring tools that support instructors in efficiently producing high-quality, interactive learning content.

Proposed method

• An authoring system was developed that takes a Java code example and its problem statement as input, then uses a pre-trained LLM (e.g., ChatGPT) to generate line-by-line explanations.
• The system presents the AI-generated explanations to instructors for review and editing, enabling human-AI collaboration in the authoring process.
• The system supports exporting examples in reusable formats for integration into existing example-exploration tools like WebEx or PCEX.
• A user study was conducted with 15 TAs and students, comparing AI-generated and expert-written explanations on completeness and quality using a 3-point scale.
• Explanations were evaluated on a 3-point scale: 'AI better', 'expert better', or 'both the same', with ratings aggregated across participants.
• The study used a single, fixed prompt for all examples to assess generalizability, though future work may explore difficulty-specific prompts.

Experimental results

Research questions

• RQ1 Can LLMs generate line-by-line code explanations for Java examples that are considered pedagogically sufficient and accurate by human evaluators?
• RQ2 How does the quality of AI-generated explanations compare to expert-written explanations in terms of completeness and clarity?
• RQ3 What is the perceived value of AI-generated explanations among instructors and teaching assistants in a human-AI collaborative authoring workflow?
• RQ4 To what extent do evaluators prefer AI-generated explanations over expert-written ones, and what factors influence this preference?
• RQ5 How can the design of prompts and user interfaces support instructors in effectively editing and refining AI-generated explanations?

Key findings

• 53.93% of evaluators rated the AI-generated explanations as better or equally complete compared to expert explanations, indicating strong perceived quality.
• On average, AI-generated explanations scored 1.867 on a 3-point completeness scale (with 0 = both the same, 1 = expert better, 2 = AI better), while expert explanations scored 1.400, showing a statistically favorable preference for AI output.
• Students rated AI explanations as better or equal in 51.11% of cases, while TAs and authors showed even higher preference at 58.15% and 57.78% respectively.
• Despite the overall positive reception, 6.96% of evaluators found the AI explanations incomplete, and 14.07% of expert explanations were rated as incomplete, highlighting room for improvement.
• The study revealed that both experts and AI sometimes chose not to explain certain lines, but this decision-making process was not evaluated, indicating a need for future research on when to explain or skip lines.
• The study's small sample size (15 participants) and limited number of examples (eight) suggest that results should be interpreted cautiously, and larger-scale studies are needed for generalization.