[Paper Review] Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study

David James Woo, Deliang Wang+2 more|arXiv (Cornell University)|Jul 30, 2024

Diverse Interdisciplinary Research Innovations5 citations

TL;DR

This study designs and tests a 100-minute prompt engineering intervention for undergraduates, examining its impact on AI self-efficacy, AI knowledge, and prompt engineering ability using mixed methods with 27 participants.

ABSTRACT

Prompt engineering is critical for effective interaction with large language models (LLMs) such as ChatGPT. However, efforts to teach this skill to students have been limited. This study designed and implemented a prompt engineering intervention, examining its influence on undergraduate students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts. The intervention involved 27 students who participated in a 100-minute workshop conducted during their history course at a university in Hong Kong. During the workshop, students were introduced to prompt engineering strategies, which they applied to plan the course's final essay task. Multiple data sources were collected, including students' responses to pre- and post-workshop questionnaires, pre- and post-workshop prompt libraries, and written reflections. The study's findings revealed that students demonstrated a higher level of AI self-efficacy, an enhanced understanding of AI concepts, and improved prompt engineering skills because of the intervention. These findings have implications for AI literacy education, as they highlight the importance of prompt engineering training for specific higher education use cases. This is a significant shift from students haphazardly and intuitively learning to engineer prompts. Through prompt engineering education, educators can faciitate students' effective navigation and leverage of LLMs to support their coursework.

Motivation & Objective

• Investigate whether a structured prompt engineering intervention improves undergraduates' AI self-efficacy.
• Examine changes in students' understanding of AI concepts after the intervention.
• Assess enhancements in students' prompt engineering ability as applied to coursework.
• Explore students' reflections to understand how the intervention influences AI literacy.

Proposed method

• Implement a 100-minute prompt engineering workshop within a university history course in Hong Kong.
• Involve 27 students and introduce prompt engineering strategies.
• Apply the strategies to plan and support the course’s final essay task.
• Collect data from pre- and post-workshop questionnaires, pre- and post-workshop prompt libraries, and written reflections.
• Analyze data using mixed methods to triangulate quantitative gains and qualitative insights.

Experimental results

Research questions

• RQ1 Does the prompt engineering intervention increase students' AI self-efficacy?
• RQ2 Does the intervention enhance students' AI knowledge and understanding of AI concepts?
• RQ3 Does the intervention improve students' prompt engineering ability in the context of coursework?
• RQ4 What insights do student reflections provide about the learning process and literacy gains?

Key findings

• Students showed higher AI self-efficacy after the intervention.
• Students demonstrated an enhanced understanding of AI concepts post-intervention.
• Students' prompt engineering skills improved as a result of the workshop.
• Qualitative reflections suggest the intervention supports more deliberate and effective navigation of LLMs in coursework.