[Paper Review] Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice

Vinay Pursnani, Yusuf Sermet+1 more|arXiv (Cornell University)|Apr 20, 2023

Artificial Intelligence in Healthcare and Education18 citations

TL;DR

The paper evaluates ChatGPT (GPT-4-based) performance on the FE Environmental Exam, showing prompt modification can significantly improve accuracy and highlighting growing mathematical capabilities across model iterations, with discussion of educational implications and future research directions.

ABSTRACT

In recent years, advancements in artificial intelligence (AI) have led to the development of large language models like GPT-4, demonstrating potential applications in various fields, including education. This study investigates the feasibility and effectiveness of using ChatGPT, a GPT-4 based model, in achieving satisfactory performance on the Fundamentals of Engineering (FE) Environmental Exam. This study further shows a significant improvement in the model's accuracy when answering FE exam questions through noninvasive prompt modifications, substantiating the utility of prompt modification as a viable approach to enhance AI performance in educational contexts. Furthermore, the findings reflect remarkable improvements in mathematical capabilities across successive iterations of ChatGPT models, showcasing their potential in solving complex engineering problems. Our paper also explores future research directions, emphasizing the importance of addressing AI challenges in education, enhancing accessibility and inclusion for diverse student populations, and developing AI-resistant exam questions to maintain examination integrity. By evaluating the performance of ChatGPT in the context of the FE Environmental Exam, this study contributes valuable insights into the potential applications and limitations of large language models in educational settings. As AI continues to evolve, these findings offer a foundation for further research into the responsible and effective integration of AI models across various disciplines, ultimately optimizing the learning experience and improving student outcomes.

Motivation & Objective

• Assess feasibility and effectiveness of using ChatGPT to take the FE Environmental Exam.
• Investigate how noninvasive prompt modifications affect AI accuracy on engineering exam questions.
• Evaluate changes in mathematical/problem-solving capabilities across ChatGPT iterations.
• Discuss educational and professional practice implications and future research directions.

Proposed method

• Apply ChatGPT (GPT-4 based) to FE Environmental Exam questions to measure initial performance.
• Implement noninvasive prompt modifications to improve response accuracy and compare performance.
• Analyze improvements in mathematical/problem-solving tasks across successive ChatGPT iterations.
• Interpret results in the context of engineering education, accessibility, and exam integrity.

Experimental results

Research questions

• RQ1 Can ChatGPT achieve satisfactory performance on the FE Environmental Exam?
• RQ2 Do noninvasive prompt modifications meaningfully improve ChatGPT accuracy on FE questions?
• RQ3 How do mathematical capabilities of ChatGPT evolve across model iterations in the FE context?
• RQ4 What are the implications for education, accessibility, and AI-resistant exam design in professional engineering practice?

Key findings

• ChatGPT shows improved accuracy on FE Environmental Exam questions with prompt modifications.
• Model iterations exhibit remarkable improvements in mathematical capabilities relevant to engineering problems.
• Prompt engineering emerges as a viable approach to enhance AI performance in educational settings.
• Findings inform potential applications and limitations of large language models in engineering education and practice.