[论文解读] Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study

David James Woo, Deliang Wang+2 more|arXiv (Cornell University)|Jul 30, 2024

Diverse Interdisciplinary Research Innovations被引用 5

一句话总结

本研究设计并测试一个 100-minute 提示工程干预，面向本科生，研究其对 AI 自我效能、AI 知识和提示工程能力的影响，采用混合方法，参与者共有 27 名。

ABSTRACT

Prompt engineering is critical for effective interaction with large language models (LLMs) such as ChatGPT. However, efforts to teach this skill to students have been limited. This study designed and implemented a prompt engineering intervention, examining its influence on undergraduate students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts. The intervention involved 27 students who participated in a 100-minute workshop conducted during their history course at a university in Hong Kong. During the workshop, students were introduced to prompt engineering strategies, which they applied to plan the course's final essay task. Multiple data sources were collected, including students' responses to pre- and post-workshop questionnaires, pre- and post-workshop prompt libraries, and written reflections. The study's findings revealed that students demonstrated a higher level of AI self-efficacy, an enhanced understanding of AI concepts, and improved prompt engineering skills because of the intervention. These findings have implications for AI literacy education, as they highlight the importance of prompt engineering training for specific higher education use cases. This is a significant shift from students haphazardly and intuitively learning to engineer prompts. Through prompt engineering education, educators can faciitate students' effective navigation and leverage of LLMs to support their coursework.

研究动机与目标

• 调查结构化的提示工程干预是否提升本科生的 AI 自我效能。
• 考察干预后学生对 AI 概念理解的变化。
• 评估在课程作业中应用的提示工程能力的提升。
• 探究学生的反思，以了解干预如何影响 AI 素养。

提出的方法

• 在香港的一门大学历史课程中实施一个 100-minute 的提示工程工作坊。
• 涉及 27 名学生并引入提示工程策略。
• 将这些策略应用于规划并支持课程的期末论文任务。
• 从工作坊前后问卷、工作坊前后提示库以及书面反思中收集数据。
• 使用混合方法对数据进行分析，以三角验证定量增益和定性洞察。

实验结果

研究问题

• RQ1 提示工程干预是否提升学生的 AI 自我效能？
• RQ2 干预是否提升学生的 AI 知识和对 AI 概念的理解？
• RQ3 在课程作业情境中，干预是否提高学生的提示工程能力？
• RQ4 学生反思对学习过程和素养提升提供了哪些见解？

主要发现

• 干预后，学生的 AI 自我效能提高。
• 干预后，学生对 AI 概念的理解得到增强。
• 工作坊使学生的提示工程技能得到提升。
• 定性反思表明该干预有助于在课程作业中更有意识且更有效地使用 LLMs。