[论文解读] Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach

Irina Jurenka, Markus Kunesch+72 more|arXiv (Cornell University)|May 21, 2024

Online Learning and Analytics被引用 17

一句话总结

本文提出 LearnLM-Tutor，一款基于 Gemini 1.0 的文本型教育 AI 导师，以及一个以评估为导向、参与式的方法论，包含七个教学基准，用以评估并提升教学能力。

ABSTRACT

A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.

研究动机与目标

• 通过开发一个负责任、以评估为中心的生成式AI导师，促进优质教育的公平获取。
• 将学习科学原理转化为 Gemini 1.0 的切实教育改进。
• 建立一个全面、多层面的评估框架，以评估AI导师的教学能力。
• 与学习者和教育者共同设计导师，使其符合现实世界的需求与约束。

提出的方法

• 通过对 Gemini 1.0 进行1对1对话式教学的微调来开发 LearnLM-Tutor（监督式微调；后续考虑 RLHF，但在本工作中未实现）。
• 创建并部署七个教学基准，覆盖定量、定性、自动和人工评估（如评估分类法所示）。
• 组装高质量的微调数据，基于参与式设计原则与教育材料（例如，共享的课程材料和视频）。
• 使用快速的自动评估循环和较慢的人类评估循环来引导迭代模型改进。
• 结合参与式设计方法（工作坊、访谈、Wizard-of-Oz 会话）与学习者和教育者共同定义目标和评估标准。

Figure 1 : LearnLM-Tutor Development : overview of our approach to responsible development of gen AI for education. Bold arrows show the development flow, dotted arrows the information flow. Our approach starts and ends with participation . We start by answering the questions of “who are we trying t

实验结果

研究问题

• RQ1 AI 导师应具备哪些核心教学能力以支持1:1教育？
• RQ2 参与式、跨学科的过程如何为教育用AI导师的开发与评估提供信息？
• RQ3 微调模型（LearnLM-Tutor）在教学基准上相比提示调优基线在多大程度上更具优势？
• RQ4 在大规模部署教育用通用AI时的伦理、安全与政策考量有哪些？

主要发现

• 在多个教学维度上，教育工作者和学习者都更偏好 LearnLM-Tutor，相对于经过提示调优的 Gemini。
• 七个基准评估框架可以涵盖AI导师教学能力的广泛范围。
• 参与式设计方法将模型改进有效地扎根于真实学习材料和学习者需求。
• 使用高质量、扎根的辅导数据进行微调，能实现比单纯提示更符合教学的行为。
• 该工作强调了教育聚焦的AI部署中需要持续关注的局限性与安全/伦理考量。

Figure 2 : Overview of the evaluation taxonomy introduced in Section 4.3.2 that underpins the seven pedagogical evaluation benchmarks introduced in this report. Each benchmark is unique in its place within the taxonomy and comes with its own benefits and challenges. Together, these different benchma