[论文解读] How critically can an AI think? A framework for evaluating the quality of thinking of generative artificial intelligence

Luke Zaphir, Jason M. Lodge+3 more|arXiv (Cornell University)|Jun 20, 2024

Explainable Artificial Intelligence (XAI)被引用 5

一句话总结

本文提出 MAGE 框架，用于评析评估对生成式 AI 的脆弱性，引导学科评估设计以评估并改进批判性思维任务。

ABSTRACT

Generative AI such as those with large language models have created opportunities for innovative assessment design practices. Due to recent technological developments, there is a need to know the limits and capabilities of generative AI in terms of simulating cognitive skills. Assessing student critical thinking skills has been a feature of assessment for time immemorial, but the demands of digital assessment create unique challenges for equity, academic integrity and assessment authorship. Educators need a framework for determining their assessments vulnerability to generative AI to inform assessment design practices. This paper presents a framework that explores the capabilities of the LLM ChatGPT4 application, which is the current industry benchmark. This paper presents the Mapping of questions, AI vulnerability testing, Grading, Evaluation (MAGE) framework to methodically critique their assessments within their own disciplinary contexts. This critique will provide specific and targeted indications of their questions vulnerabilities in terms of the critical thinking skills. This can go on to form the basis of assessment design for their tasks.

研究动机与目标

• 激发评估生成式 AI 在模拟认知技能方面的极限与能力的必要性。
• 提供一个在学科背景内系统性评析评估的框架。
• 提供可操作的设计评估的方法，以提高对 AI 生成的鲁棒性。
• 在数字化评估中解决公平性、学术诚信和评估作者身份等问题。

提出的方法

• 提出 MAGE 框架：问题映射、AI 脆弱性测试、评分、评估。
• 以 ChatGPT-4 作为当前行业基准来测试 AI 在批判性思维任务中的能力。
• 概述将问题映射到潜在 AI 脆弱性并对回答进行评分与评估的步骤。
• 提供针对学科背景的具体解读 AI 脆弱性发现的指导。

实验结果

研究问题

• RQ1 如何系统性地评析评估对生成式 AI（如 ChatGPT-4）的易感性？
• RQ2 哪些指标揭示 AI 在学科任务中的批判性思维质量？
• RQ3 MAGE 框架如何为评估设计提供信息以降低 AI 脆弱性？
• RQ4 在数字评估情境中有哪些关于公平性、诚信和署名权的考量？

主要发现

• 本文提供 MAGE 框架作为评析评估对 AI 脆弱性的方法。
• 该框架能够就批判性思维技能方面对问题的脆弱性进行精准指示。
• 该方法支持在学科背景内改进评估设计。
• 该框架凸显数字化评估中与公平性、学术诚信和署名权相关的关注点。