[论文解读] The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers

James Prather, Brent N. Reeves+7 more|arXiv (Cornell University)|May 28, 2024

Artificial Intelligence in Healthcare and Education被引用 5

一句话总结

该研究在前一项新手编程元认知研究的基础上加入 GenAI 工具，发现 GenAI 既能帮助新手，也可能造成阻碍，同时出现新的元认知困难，并在高成就与低成就学生之间拉大数字鸿沟。

ABSTRACT

Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who accelerated and students who struggled. Students who accelerated were able to use GenAI to create code they already intended to make and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.

研究动机与目标

• 调查 GenAI 工具如何影响新手程序员解决编程问题的过程。
• 检验 GenAI 是否会加剧先前已识别的元认知困难。
• 识别由 GenAI 使用引发的任何新元认知困难。
• 评估 GenAI 使用与学生表现及自我效能感之间的关系。

提出的方法

• 重复前一项实验室研究，使用相同的问题和集成到 Canvas 的自动评估工具（Athene）。
• 在 CS1 课程中，学生在解决问题时使用 VSCode、GitHub Copilot 和 ChatGPT。
• 通过参与者观察、思考大声表达、眼动追踪（Tobii）、访谈和自我效能量表来收集数据。
• 将数据按既定的元认知阶段与困难的编码手册进行编码，并为 GenAI 特定问题新增编码。
• 分析元认知困难、任务时长、成绩和自我效能感之间的相关性。

Figure 1. Problem description from Athene

实验结果

研究问题

• RQ1 RQ1：新手程序员在使用 GenAI 工具解决编程问题时能获得哪些收益？
• RQ2 RQ2：新手程序员在使用 GenAI 工具解决编程问题时会遇到哪些困难？

主要发现

• 使用 GenAI 使部分学生更快接近解决方案，而其他学生则表现出持续的元认知困难。
• 出现新的 GenAI 特定元认知困难，包括 Interruption、Mislead 和 Progression，以及现有的困难。
• 成绩较高且自我效能感更强的学生倾向于使用 GenAI 以加速解决问题，而成绩较低的学生则表现出能力幻觉。
• 出现元认知困难的学生往往更频繁地接受 Copilot 的建议，有时会在后续重新加工或舍弃它们。
• 研究观察到 GenAI 的益处与危害并存，有迹象表明 GenAI 可能扩大成就差距并强化对挣扎学生的既有挑战。

Figure 2. Sample Feedback from Athene