[论文解读] Can AI Chatbots Pass the Fundamentals of Engineering (FE) and Principles and Practice of Engineering (PE) Structural Exams?

M. Z. Naser, Brandon E. Ross+5 more|arXiv (Cornell University)|Mar 31, 2023

AI in Service Interactions被引用 8

一句话总结

本论文评估 AI 聊天机器人 ChatGPT-4 与 Google Bard 是否能够通过 FE 与 PE Structural 考试，报告了接近及通过的分数表现，并讨论对教学与工程指导的启示。

ABSTRACT

The engineering community has recently witnessed the emergence of chatbot technology with the release of OpenAI ChatGPT-4 and Google Bard. While these chatbots have been reported to perform well and even pass various standardized tests, including medical and law exams, this forum paper explores whether these chatbots can also pass the Fundamentals of Engineering (FE) and Principles and Practice of Engineering (PE) exams. A diverse range of civil and environmental engineering questions and scenarios are used to evaluate the chatbots' performance, as commonly present in the FE and PE exams. The chatbots' responses were analyzed based on their relevance, accuracy, and clarity and then compared against the recommendations of the National Council of Examiners for Engineering and Surveying (NCEES). Our report shows that ChatGPT-4 and Bard, respectively scored 70.9% and 39.2% in the FE exam and 46.2% and 41% in the PE exam. It is evident that the current version of ChatGPT-4 could potentially pass the FE exam. While future editions are much more likely to pass both exams, this study also highlights the potential of using chatbots as teaching assistants and guiding engineers.

研究动机与目标

• 评估现代 AI 聊天机器人是否能够通过土木/环境工程领域的 FE 与 PE Structural 考试。
• 量化聊天机器人表现并与 NCEES 的建议进行比较。
• 讨论对工程教育及 AI 工具在导师角色中的影响。

提出的方法

• 编制代表 FE/PE 考试的多样化土木/环境工程问题和场景。
• 评估 ChatGPT-4 与 Bard 的回答在相关性、准确性和清晰度方面的表现。
• 将聊天机器人表现与 NCEES 的建议及共识期望进行比较。

实验结果

研究问题

• RQ1 ChatGPT-4 与 Bard 是否在 FE 与 PE Structural 考试上达到类似通过的分数？
• RQ2 聊天机器人回答在准确性和清晰度方面与 NCEES 的建议是否一致？
• RQ3 聊天机器人表现对教学和工程实践有何影响？

主要发现

• ChatGPT-4 在 FE 考试得分为 70.9%，Bard 得分 39.2%。
• 在 PE 考试中，ChatGPT-4 得分 46.2%，Bard 得分 41%。
• 结果表明在当前评估条件下，ChatGPT-4 可能具备通过 FE 考试的潜力。
• 研究强调聊天机器人作为教学助理和工程师指导工具的潜在角色。