[论文解读] Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators

Paul Denny, Juho Leinonen+5 more|arXiv (Cornell University)|Jul 31, 2023

Software Engineering Research被引用 15

一句话总结

本论文引入 Prompt Problems 作为教学法，教授学生为代码生成型大语言模型（LLMs）设计有效提示，并提出 Promptly —— 一个在一年级 Python 课程中提供并自动评估提示生成代码的网页工具（n=54）。

ABSTRACT

With their remarkable ability to generate code, large language models (LLMs) are a transformative technology for computing education practice. They have created an urgent need for educators to rethink pedagogical approaches and teaching strategies for newly emerging skill sets. Traditional approaches to learning programming have focused on frequent and repeated practice at writing code. The ease with which code can now be generated has resulted in a shift in focus towards reading, understanding and evaluating LLM-generated code. In parallel with this shift, a new essential skill is emerging -- the ability to construct good prompts for code-generating models. This paper introduces a novel pedagogical concept known as a `Prompt Problem', designed to help students learn how to craft effective prompts for LLMs. A Prompt Problem challenges a student to create a natural language prompt that leads an LLM to produce the correct code for a specific problem. To support the delivery of Prompt Problems at scale, in this paper we also present a novel tool called Promptly which hosts a repository of Prompt Problems and automates the evaluation of prompt-generated code. We report empirical findings from a field study in which Promptly was deployed in a first-year Python programming course (n=54). We explore student interactions with the tool and their perceptions of the Prompt Problem concept. We found that Promptly was largely well-received by students for its ability to engage their computational thinking skills and expose them to new programming constructs. We also discuss avenues for future work, including variations on the design of Prompt Problems and the need to study their integration into the curriculum and teaching practice.

研究动机与目标

• 在LLM生成代码的时代，激励将提示工程与传统编程并行教授的必要性。
• 引入 Prompt Problem 的概念：学生用自然语言提示来生成正确的代码。
• 描述 Promptly 工具：提供 Prompt Problems 并自动评估生成的代码。
• 报告在一年级 Python 课程中的现场部署的实证发现，并反思学生体验。
• 讨论将 Prompt Problems 融入课程的设计选项及未来方向。

提出的方法

• 将 Prompt Problem 定义并举例为一个练习：给出一个可视化问题，学生设计提示以引导 LLM 生成所需代码。
• 开发 Promptly，一款基于网页的工具（React/NodeJS），展示问题视觉、接受学生提示、调用 LLM API，并在沙箱中对生成的代码进行测试用例运行。
• 在一年级 Python 课程中将 Promptly 部署为一个大型、可选的实验任务（n=54），以观察互动、成功率和学生认知。
• 收集定量数据（平均提交数、成功计数、提示单词数）和定性反馈，以了解学习者体验及感知的益处/担忧。
• 讨论 Prompt Problem 的变体、整合到课程的方式，以及未来研究的方向。

实验结果

研究问题

• RQ1 RQ1：学生在使用 Promptly 工具时在总体成功率和提示长度方面的互动是怎样的？
• RQ2 RQ2：学生对 Promptly 工具以及通过构建提示来学习编程的看法如何？

主要发现

• Promptly 在解决问题时获得了广泛好评，并让学生参与到计算思维中。
• 问题3对学生来说最难，平均提交数最高（6.4），解题人数较少（19），与问题1的43名解题者及问题2的32名解题者相比。
• 各问题的平均提示长度不同，分别为问题1、2、3的13、38、36个单词。
• 研究发现让学生接触新的编程结构，以及提升他们对问题规范和提示设计的批判性思维能力的好处。
• 一些学生表达了对依赖 AI 工具的抵触或担忧，并强调负责任使用及保持编程独立性的重要性。
• 反思强调提示工程代表一种不同形式的编程，需要显式的教学和练习。