Skip to main content
QUICK REVIEW

[论文解读] Investigating the Effectiveness of ChatGPT in Mathematical Reasoning and Problem Solving: Evidence from the Vietnamese National High School Graduation Examination

Xuan-Quy Dao, Ngoc-Bich Le|arXiv (Cornell University)|Jun 10, 2023
Online Learning and AnalyticsComputer Science被引用 52
一句话总结

该研究评估 ChatGPT 在 250 道越南 VNHSGE 数学题目中在四个难度等级和十个主题上的表现,揭示在知识层面题目表现强劲,但难度增加时下降,且在特定主题方面有显著优势和劣势。

ABSTRACT

This study offers a complete analysis of ChatGPT's mathematics abilities in responding to multiple-choice questions for the Vietnamese National High School Graduation Examination (VNHSGE) on a range of subjects and difficulty levels. The dataset included 250 questions divided into four levels: knowledge (K), comprehension (C), application (A), and high application (H), and it included ten themes that covered diverse mathematical concepts. The outcomes demonstrate that ChatGPT's performance varies depending on the difficulty level and subject. It performed best on questions at Level (K), with an accuracy rate of $83\%$; but, as the difficulty level rose, it scored poorly, with an accuracy rate of $10\%$. The study has also shown that ChatGPT significantly succeeds in providing responses to questions on subjects including exponential and logarithmic functions, geometric progression, and arithmetic progression. The study found that ChatGPT had difficulty correctly answering questions on topics including derivatives and applications, spatial geometry, and Oxyz spatial calculus. Additionally, this study contrasted ChatGPT outcomes with Vietnamese students in VNHSGE and in other math competitions. ChatGPT dominated in the SAT Math competition with a success rate of $70\%$, followed by VNHSGE mathematics ($58.8\%)$. However, its success rates were lower on other exams, such as AP Statistics, the GRE Quantitative, AMC 10, AMC 12, and AP Calculus BC. These results suggest that ChatGPT has the potential to be an effective teaching tool for mathematics, but more work is needed to enhance its handling of graphical data and address the challenges presented by questions that are getting more challenging.

研究动机与目标

  • 使用 VNHSGE 数学数据集(2019–2023)评估 ChatGPT 的数学推理和问题求解能力。
  • 识别性能如何随题目难度、主题和形式变化。
  • 将 ChatGPT 的结果与教育基准和其他考试进行比较,以评估其作为教学/评估辅助工具的潜力。

提出的方法

  • 收集官方 VNHSGE 数学题(2019–2023)并将其分为四个等级(K、C、A、H)和十个主题(M11A–M12G)。
  • 将题目转换为 LaTeX,然后再转为 JSON 以输入给 ChatGPT;获取 ChatGPT 输出并与官方答案比较以计算准确性。
  • 按等级、主题和年份分析表现;可视化分布并计算年度平均分(例如 2019–2023)。
  • 构建知识矩阵(Table 1)和分布图(Figure 1)以描述题目结构和覆盖范围。
  • 使用结构化的提示/响应格式以确保 ChatGPT 输出的一致性(Table 2、Figure 2)。
Figure 1: Distribution of the number of questions by levels and topics in percentage.
Figure 1: Distribution of the number of questions by levels and topics in percentage.

实验结果

研究问题

  • RQ1ChatGPT 在不同认知水平(K、C、A、H)上的越南高中数学题表现如何?
  • RQ2哪些主题(M11A–M12G)在 ChatGPT 的准确性上表现较高或较低,主题与难度的交互如何影响结果?
  • RQ3ChatGPT 的在 2019–2023 年的表现趋势如何反映学习或模型改进?
  • RQ4这些结果对将 ChatGPT 作为数学教育中的教学/评估工具有何意义?

主要发现

  • 2019–2023 的总体平均分为 5.88/10(147/250)。
  • ChatGPT 在 K 水平题目上的准确率最高(各年 75–90%),在 C、A、H 水平下降,分别为 40–72%、0–57%、0–22%。
  • ChatGPT 在若干主题(如 M11B)多年来呈现 100% 的准确度,但因主题差异很大(如 M12G 0–75%),且在许多年份错过 H 水平表现。
  • 随着题目难度的增加,表现下降,表明在将概念应用于新颖或复杂问题上存在局限。
  • 某些主题(如 M11A、M11B、M12D、M12F)表现强劲,而像求导和应用(M12A)以及 M12G 的表现较弱或不稳定。
  • 题目顺序会影响 ChatGPT 的表现,暗示可能有疲劳样效应,并对考试设计有影响。
Figure 6: Distribution of the percentage of correct answer in levels and topics.
Figure 6: Distribution of the percentage of correct answer in levels and topics.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。