[论文解读] Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code

Jaromír Šavelka, Arav Agarwal+2 more|arXiv (Cornell University)|Mar 9, 2023

Software Engineering Research被引用 4

一句话总结

本研究评估了GPT模型在包含Python入门与中级课程代码片段的多项选择题（MCQ）中的表现。尽管在自然语言MCQ上表现良好（准确率为77.9%），GPT模型在代码相关推理任务中表现显著不足，代码相关MCQ的准确率仅为59.5%，尤其在需要执行追踪或逻辑分析的问题上表现最差（准确率为56.6%）。

ABSTRACT

We analyzed effectiveness of three generative pre-trained transformer (GPT) models in answering multiple-choice question (MCQ) assessments, often involving short snippets of code, from introductory and intermediate programming courses at the postsecondary level. This emerging technology stirs countless discussions of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming education (e.g., cheating). However, the capabilities of GPT models and their limitations to reason about and/or analyze code in educational settings have been under-explored. We evaluated several OpenAI's GPT models on formative and summative MCQ assessments from three Python courses (530 questions). We found that MCQs containing code snippets are not answered as successfully as those that only contain natural language. While questions requiring to fill-in a blank in the code or completing a natural language statement about the snippet are handled rather successfully, MCQs that require analysis and/or reasoning about the code (e.g., what is true/false about the snippet, or what is its output) appear to be the most challenging. These findings can be leveraged by educators to adapt their instructional practices and assessments in programming courses, so that GPT becomes a valuable assistant for a learner as opposed to a source of confusion and/or potential hindrance in the learning process.

研究动机与目标

• 调查GPT模型在编程教育中回答包含代码片段的多项选择题（MCQ）的有效性。
• 识别对GPT模型最具挑战性的MCQ类型，特别是需要对代码行为或执行过程进行推理的问题。
• 比较不同GPT模型变体（text-davinci-001、text-davinci-002、text-davinci-003）在代码相关测评中的表现。
• 为教育工作者提供实证洞察，以负责任地将GPT工具整合到编程教学与评估中。
• 为理解大语言模型在程序语义推理方面的局限性（尤其是执行追踪与逻辑属性）奠定基础。

提出的方法

• 从三门本科Python课程中收集了530道MCQ，经人工整理并按题型分类。
• 根据认知需求将题目分类为真假题、输出预测题、填空题等类型。
• 使用零样本提示法对三种GPT模型（text-davinci-001、text-davinci-002、text-davinci-003）在相同MCQ集上进行评估。
• 采用精确匹配和Jaccard相似度指标衡量性能，随机猜测的基线Jaccard相似度为23.4%。
• 通过模式匹配与人工验证确保标签一致性与性能测量的可靠性。
• 分析不同题型间的性能差异，以识别认知挑战，如执行模拟与意图推断。

Figure 1: The upper-left screenshot depicts a conversation with ChatGPT when asked to explain a code snippet line by line. It correctly explains the bahavior (1). The lower-right shows a conversation with ChatGPT when asked to generate the code snippet with the same behavior. The generated code is c

实验结果

研究问题

• RQ1 在包含代码片段的MCQ与仅含自然语言的MCQ之间，GPT模型的表现是否存在显著差异？
• RQ2 在代码推理任务中，哪些类型的MCQ（如输出预测、真假题、填空题）对GPT模型最具挑战性？
• RQ3 相较于旧版本（如text-davinci-001），新型GPT模型（如text-davinci-003）在代码相关推理任务中的表现如何？
• RQ4 GPT模型在多大程度上无法模拟代码执行或推理程序行为，这种缺陷如何影响其准确率？

主要发现

• 表现最出色的GPT模型text-davinci-003在530道MCQ上整体准确率达到65.5%，显著高于23.4%的Jaccard相似度基线。
• 包含代码片段的MCQ准确率为59.5%，远低于仅含自然语言MCQ的77.9%准确率。
• 填空题或句子补全题处理得最好（准确率为87.1%），表明其具备强大的模式匹配与补全能力。
• 需要对代码行为进行推理的题目（如预测输出或判断执行过程的真假陈述）准确率仅为56.6%，表明其在执行模拟方面存在核心局限。
• 代码题与非代码题之间的性能差距表明，GPT模型在模拟控制流与数据流方面存在困难，即使其能正确解释或生成代码。
• 尽管在代码解释与生成方面表现良好，GPT模型在需要基于追踪的推理任务中仍表现不佳，凸显其在代码功能理解上的关键差距。

Figure 2: Distribution of MCQs into categories. Note that the MCQs asking about the output of a code snippet as well as MCQs focused on filling-in the blanks in a snippet are not present in the MCQs with no code. This is to be expected given the nature of those questions. The MCQs with code are quit