[Paper Review] ChatGPT in the Classroom: Boon or Bane for Physics Students' Academic Performance?

Manuel G. Forero, H. J. Herrera-Suárez|arXiv (Cornell University)|Dec 5, 2023

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

This experimental study examines the impact of ChatGPT on physics students' academic performance, comparing a control group using traditional methods with an experimental group using ChatGPT as a learning aid. Results show a significant decline in performance, with lower grades and negative Hake factors, despite students reporting improved conceptual understanding and tool adaptability, highlighting risks of reduced critical thinking and overreliance on AI-generated content.

ABSTRACT

This study investigates the influence of ChatGPT, an AI-based language model, on student performance in a physics course. We conducted an experimental analysis with two cohorts of students in a second-semester engineering physics course. The control group (Physics 2 2022B) used traditional teaching methods, while the experimental group (Physics 2 2023A) integrated ChatGPT as a learning tool. Our results indicate that the use of ChatGPT led to a significant decrease in student performance, as evidenced by lower grades and negative Hake factors compared to the control group. In addition, a survey of students revealed conflicting perceptions of the usefulness of ChatGPT in teaching physics. While most recognized its usefulness in understanding concepts and providing information, concerns were raised about its potential to reduce critical thinking and independent learning. These findings suggest that while ChatGPT can be a useful tool, it should be used with caution and as a supplement to traditional teaching methods, rather than as a stand-alone solution. The study underlines the importance of critical and reflective use of AI tools in educational settings and highlights the irreplaceable role of teachers in providing comprehensive educational support.

Motivation & Objective

• To assess the effect of integrating ChatGPT as a learning tool on academic performance in a second-semester engineering physics course.
• To compare student performance between a control group using traditional teaching methods and an experimental group using ChatGPT.
• To investigate students' perceptions of ChatGPT’s usefulness, accuracy, and impact on critical thinking and independent learning.
• To evaluate whether AI tools like ChatGPT enhance or hinder deep learning and conceptual understanding in physics education.
• To inform educational strategies that balance AI tool integration with the preservation of critical thinking and teacher-led instruction.

Proposed method

• Two cohorts of engineering physics students were studied: one in 2022B (control group) using traditional teaching methods without ChatGPT, and another in 2023A (experimental group) with mandatory ChatGPT integration.
• The control group received instruction via lectures, blackboards, slides, videos, simulations, and lab sessions, following the standard syllabus and textbook.
• The experimental group was trained to use ChatGPT 3.5 from the start and encouraged to use it for concept clarification, problem generation, and homework support.
• Performance was measured through standardized exams, with the Hake factor used to assess normalized gain in conceptual understanding.
• A survey was administered to assess student perceptions regarding ChatGPT’s usefulness, accuracy, adaptability, and impact on independent learning.
• Statistical comparison of exam scores and Hake factors between groups was conducted to evaluate academic performance differences.

Experimental results

Research questions

• RQ1 Does the integration of ChatGPT into physics instruction lead to improved academic performance compared to traditional teaching methods?
• RQ2 How does ChatGPT usage affect students’ conceptual understanding, as measured by normalized gain (Hake factor)?
• RQ3 What are students’ perceptions of ChatGPT’s usefulness in understanding physics concepts and supporting independent learning?
• RQ4 To what extent does ChatGPT usage correlate with reduced critical thinking or overreliance on AI-generated responses?
• RQ5 Can ChatGPT serve as an effective supplement to traditional instruction without compromising learning outcomes?

Key findings

• Students in the ChatGPT-integrated group (Physics 2 2023A) earned significantly lower exam grades compared to the control group (Physics 2 2022B).
• The Hake factor, a measure of normalized conceptual gain, was negative for all exams in the experimental group, indicating a decline in conceptual understanding.
• Over 60% of students reported that interacting with ChatGPT helped them deepen their understanding of physics concepts.
• 68% of students were satisfied with ChatGPT’s ability to adapt to their knowledge level and respond to specific questions.
• Nearly 30% of students reported that using ChatGPT made them less likely to think critically or independently.
• More than 10% of students expressed dissatisfaction with ChatGPT’s accuracy or clarity in explanations.