[論文レビュー] CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes

Mark Liffiton, Brad E. Sheese+2 more|arXiv (Cornell University)|Aug 14, 2023

Software Engineering Research被引用数 9

ひとこと要約

CodeHelp は guardrails を備えた LLM 搭載のチュータリングツールで、オンデマンドのプログラミング支援をスケールで提供し、 evaluated in a 52-student, 12-week first-year course において評価された。 評価は好評で、講師サポートを補完した。

ABSTRACT

Computing educators face significant challenges in providing timely support to students, especially in large class settings. Large language models (LLMs) have emerged recently and show great promise for providing on-demand help at a large scale, but there are concerns that students may over-rely on the outputs produced by these models. In this paper, we introduce CodeHelp, a novel LLM-powered tool designed with guardrails to provide on-demand assistance to programming students without directly revealing solutions. We detail the design of the tool, which incorporates a number of useful features for instructors, and elaborate on the pipeline of prompting strategies we use to ensure generated outputs are suitable for students. To evaluate CodeHelp, we deployed it in a first-year computer and data science course with 52 students and collected student interactions over a 12-week period. We examine students' usage patterns and perceptions of the tool, and we report reflections from the course instructor and a series of recommendations for classroom use. Our findings suggest that CodeHelp is well-received by students who especially value its availability and help with resolving errors, and that for instructors it is easy to deploy and complements, rather than replaces, the support that they provide to students.

研究の動機と目的

• Address scalability challenges in providing timely programming support in large classes.
• Design guardrails to prevent direct solution reveals while guiding students toward learning.
• Provide instructor-facing features to observe, manage, and assess student engagement with CodeHelp.
• Demonstrate classroom deployment and gather student perceptions and usage analytics.

提案手法

• Develop a web-based tool with a simple student interface for structured help requests (language, code, error, issue).
• Implement a multi-prompt LLM workflow (sufficiency check, main response, code removal) to generate safe, educational outputs.
• Use chain-of-thought prompting in the sufficiency check to determine if clarification is needed.
• Generate two main response completions, select the higher quality, and perform code removal if needed.
• Integrate with LMS via LTI for seamless instructor access and class configuration (avoid sets of keywords).
• Collect usage data and survey responses to evaluate classroom impact and student perceptions.

実験結果

リサーチクエスチョン

• RQ1 How do students use CodeHelp across a 12-week introductory programming course?
• RQ2 What kinds of help requests do students submit and how does CodeHelp respond to them?
• RQ3 How do students perceive the usefulness, availability, and learning value of CodeHelp?
• RQ4 How does CodeHelp affect instructor workload and classroom support dynamics?

主な発見

• CodeHelp is well-received by students and helps resolve errors, with most respondents indicating value and willingness to reuse in future courses.
• Usage is consistent across the semester, with about half the class using CodeHelp weekly and a substantial subset submitting many queries.
• Most students felt CodeHelp helped them complete work (71% Agree; 9% Strongly Agree) and learn course material (56% Agree; 7% Strongly Agree).
• Instructors found CodeHelp easy to deploy and viewed it as a complement to, not a replacement for, human support.
• The tool enables ongoing access outside of class and provides instructor views to monitor queries and responses.