[论文解读] Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice

Vinay Pursnani, Yusuf Sermet+1 more|arXiv (Cornell University)|Apr 20, 2023

Artificial Intelligence in Healthcare and Education被引用 18

一句话总结

本论文评估 ChatGPT (GPT-4-based) 在 FE Environmental Exam 上的表现，显示提示修改可以显著提高准确性，并突出模型迭代中日益增长的数学能力，并对教育影响与未来研究方向进行讨论。

ABSTRACT

In recent years, advancements in artificial intelligence (AI) have led to the development of large language models like GPT-4, demonstrating potential applications in various fields, including education. This study investigates the feasibility and effectiveness of using ChatGPT, a GPT-4 based model, in achieving satisfactory performance on the Fundamentals of Engineering (FE) Environmental Exam. This study further shows a significant improvement in the model's accuracy when answering FE exam questions through noninvasive prompt modifications, substantiating the utility of prompt modification as a viable approach to enhance AI performance in educational contexts. Furthermore, the findings reflect remarkable improvements in mathematical capabilities across successive iterations of ChatGPT models, showcasing their potential in solving complex engineering problems. Our paper also explores future research directions, emphasizing the importance of addressing AI challenges in education, enhancing accessibility and inclusion for diverse student populations, and developing AI-resistant exam questions to maintain examination integrity. By evaluating the performance of ChatGPT in the context of the FE Environmental Exam, this study contributes valuable insights into the potential applications and limitations of large language models in educational settings. As AI continues to evolve, these findings offer a foundation for further research into the responsible and effective integration of AI models across various disciplines, ultimately optimizing the learning experience and improving student outcomes.

研究动机与目标

• 评估使用 ChatGPT 来参加 FE Environmental Exam 的可行性与有效性。
• 研究非侵入式提示修改如何影响 AI 在工程考试题目上的准确性。
• 评估在不同 ChatGPT 版本中数学/解题能力的变化。
• 讨论教育与专业实践的含义以及未来研究方向。

提出的方法

• 将 ChatGPT（基于 GPT-4）应用于 FE Environmental Exam 的题目以衡量初始表现。
• 实施非侵入式提示修改以提高回答的准确性并比较表现。
• 分析在连续的 ChatGPT 迭代中数学/解题任务的改进。
• 在工程教育、可访问性和考试完整性的背景下解释结果。

实验结果

研究问题

• RQ1 ChatGPT 能否在 FE Environmental Exam 上达到令人满意的表现？
• RQ2 非侵入式提示修改是否能显著提高 ChatGPT 在 FE 问题上的准确性？
• RQ3 在 FE 情境中，ChatGPT 的数学能力如何在模型迭代中演变？
• RQ4 对教育、可访问性以及面向专业工程实践的抗 AI 考试设计有哪些影响？

主要发现

• 在提示修改后，ChatGPT 对 FE Environmental Exam 问题的准确性有所提高。
• 模型迭代在与工程问题相关的数学能力方面呈现显著改进。
• 提示工程成为提升 AI 在教育环境中表现的可行方法。
• 研究结果为大型语言模型在工程教育与实践中的潜在应用与局限性提供了参考。