[Paper Review] A Preliminary Exploration of the Disruption of a Generative AI Systems: Faculty/Staff and Student Perceptions of ChatGPT and its Capability of Completing Undergraduate Engineering Coursework

Lance White, Trini Balart+3 more|arXiv (Cornell University)|Mar 3, 2024

Artificial Intelligence in Healthcare and Education5 citations

TL;DR

The paper investigates ChatGPT’s ability to complete undergraduate engineering coursework and explores faculty, staff, and student perceptions, using the DANCE model to guide faculty adaptation amid disruption.

ABSTRACT

The authors of this study aim to assess the capabilities of the OpenAI ChatGPT tool to understand just how effective such a system might be for students to utilize in their studies as well as deepen understanding of faculty/staff and student perceptions about ChatGPT in general. The purpose of what is learned from the study is to continue the design of a model to facilitate the development of faculty for becoming adept at embracing change, the DANCE model (Designing Adaptations for the Next Changes in Education). This model is used in this study to help faculty with examining the impact that a disruptive new tool, such as ChatGPT, can pose for the learning environment. The authors analyzed the performance of ChatGPT used to complete course assignments at a variety of levels by novice engineering students working as research assistants. Those completed works have been assessed by the faculty who created those assignments to understand how these completed assignments might compare with the performance of a typical student. A set of surveys conducted by the authors of this work are discussed where students, faculty, and staff respondents in March of 2023 addressed their perceptions of ChatGPT (A follow-up survey is being administered now, February 2024). These survey instruments were analyzed, and the data visualized in this work to bring attention to relevant findings by the researchers. This work reports the findings of the researchers with the purpose of sharing the current state of this work at Texas A&M University with the intention to provide insights to scholars both at our own institution and around the world. This work is not intended to be a finished work but reports these findings with full transparency that this work is currently continuing as the researchers gather new data and develop and validate various measurement instruments.

Motivation & Objective

• Assess the capabilities of OpenAI's ChatGPT to complete engineering course assignments at multiple levels using student research assistants.
• Understand faculty, staff, and student perceptions of ChatGPT and its impact on learning environments.
• Utilize and further develop the DANCE model (Designing Adaptations for the Next Changes in Education) to guide faculty preparation for disruptive tools.
• Share current findings from Texas A&M University to support scholars globally while data collection continues.

Proposed method

• Evaluate ChatGPT-generated coursework produced by novice engineering student research assistants across course levels.
• Have faculty who created the assignments assess the ChatGPT outputs to compare with typical student performance.
• Conduct surveys with students, faculty, and staff (March 2023) to capture perceptions of ChatGPT; a follow-up survey was administered in February 2024.
• Apply the DANCE model to frame analysis of disruption and adaptation needs for education.

Experimental results

Research questions

• RQ1 What is ChatGPT’s capability to complete undergraduate engineering course assignments as observed by faculty and staff?
• RQ2 How do students, faculty, and staff perceive ChatGPT and its influence on the learning environment?
• RQ3 How can the DANCE model be used to design adaptations for educators facing disruption from generative AI tools?
• RQ4 What insights emerge from comparing ChatGPT-produced work to typical student performance on the same assignments?
• RQ5 What changes are anticipated as ongoing data collection and instrument validation continue at the institution?

Key findings

• Preliminary findings highlight notable perceptions about ChatGPT and its disruptive potential in engineering education.
• ChatGPT outputs were examined relative to typical student performance by the assignment authors, offering early insights into capabilities and limitations.
• survey data visualization emphasizes meaningful trends and themes regarding adoption, concern, and instructional impact.
• The study explicitly states it is not a finished work and that data collection and instrument validation are ongoing.
• The work contributes to a broader effort at Texas A&M University to share early insights with the global scholarly community.