[Paper Review] Enhancing Physics Learning with ChatGPT, Bing Chat, and Bard as Agents-to-Think-With: A Comparative Case Study

Renato P. dos Santos|arXiv (Cornell University)|Jun 1, 2023

Artificial Intelligence in Healthcare and Education8 citations

TL;DR

A single-case study comparing four GenAI chatbots (ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard) as agents-to-think-with in physics learning, analyzing interaction logs with a simulated student to assess empathy, subject knowledge, and learning-process understanding.

ABSTRACT

The rise of AI has brought remarkable advancements in education, with AI models demonstrating their ability to analyse and provide instructive solutions to complex problems. This study compared and analysed the responses of four Generative AI-powered chatbots (GenAIbots) - ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard - within the constructivist theoretical framework. Using a single-case study methodology, interaction logs between the GenAIbots and a simulated student in Physics learning scenarios were analysed. The GenAIbots were presented with conceptually dense Physics problems to promote deep understanding. The qualitative analysis focused on tutor traits such as subject-matter knowledge, empathy, assessment emphasis, facilitation skills, and comprehension of the learning process. Findings showed that all GenAIbots functioned as agents-to-think-with, fostering critical thinking, problem-solving, and subject-matter knowledge. ChatGPT-4 stood out for demonstrating empathy and a deep understanding of the learning process. However, inconsistencies and shortcomings were observed, highlighting the need for human intervention in AI-assisted learning. In conclusion, while GenAIbots have limitations, their potential as agents-to-think-with in Physics education offers promising prospects for revolutionising instruction.

Motivation & Objective

• Investigate how GenAI-powered chatbots function as agents-to-think-with in physics learning within a constructivist framework.
• Analyse interaction logs between four GenAIbots and a simulated physics student to assess tutoring traits.
• Identify strengths and limitations of each chatbot in promoting critical thinking, problem solving, and subject-matter knowledge.
• Highlight implications for human intervention and the potential of GenAIbots to transform physics instruction.

Proposed method

• Single-case study using interaction logs between GenAIbots and a simulated student in physics learning scenarios.
• Qualitative analysis focusing on tutor traits: subject-matter knowledge, empathy, assessment emphasis, facilitation skills, and understanding of the learning process.
• Comparison across four GenAIbots: ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard.
• Assessment of their ability to promote deep understanding through conceptually dense physics problems.

Experimental results

Research questions

• RQ1 How do ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard function as agents-to-think-with in physics learning?
• RQ2 Which tutor traits (knowledge, empathy, assessment emphasis, facilitation, learning-process comprehension) are exhibited by each GenAIbot?
• RQ3 What are the strengths and limitations of each GenAIbot in fostering critical thinking and problem solving in physics?
• RQ4 To what extent is human intervention needed to support AI-assisted physics instruction?

Key findings

• All GenAIbots functioned as agents-to-think-with, fostering critical thinking, problem-solving, and subject-matter knowledge.
• ChatGPT-4 stood out for demonstrating empathy and a deep understanding of the learning process.
• Inconsistencies and shortcomings were observed across bots, indicating the need for human oversight in AI-assisted learning.
• GenAIbots show potential to revolutionize physics instruction, but require consideration of limitations and appropriate human support.