[Paper Review] Enhancing Chemistry Learning with ChatGPT, Bing Chat, Bard, and Claude as Agents-to-Think-With: A Comparative Case Study

Renato P. dos Santos|arXiv (Cornell University)|Oct 23, 2023

Online Learning and Analytics4 citations

TL;DR

This comparative case study evaluates ChatGPT, Bing Chat, Bard, and Claude as 'agents-to-think-with' in chemistry education, using simulated student interactions and content analysis. ChatGPT outperformed others in contextual comprehension, critical thinking, and Socratic questioning, demonstrating superior potential for enhancing learning through dynamic, personalized dialogue.

ABSTRACT

This research delves into the comparative advantages of Generative AI chatbots (GenAIbots) -- ChatGPT, Bing Chat, Bard, and Claude -- in the context of Chemistry education, framed within a constructivist perspective. Our primary objective was to identify which of these four AI tools is more effective for enhancing Chemistry learning. Employing a single-case study approach, we scrutinised interaction logs between the AI systems and a simulated student persona during Chemistry learning simulations, incorporating Content Analysis methodology to delve deeper into the discourse. Our findings underscore these tools' potential as "agents-to-think-with", enhancing critical thinking, problem-solving, comprehension, creativity, and tailored learning. Especially noteworthy is their ability to stimulate learners through Socratic-like questioning, aligning with constructionist principles. The research emphasises the pivotal role of prompt crafting to coax desired responses from GenAIbots, engendering iterative reflections. It also highlights the need for robust educator training to infuse these technologies into educational settings. Conclusively, while ChatGPT, Bing Chat, Bard, and Claude are poised to enrich Chemistry education by fostering dynamic, inclusive learning experiences, ChatGPT stood out, decisively surpassing Bing Chat in its performance. Bard and Claude trailed closely, with all three showcasing a more in-depth, precise, and nuanced understanding, underscoring ChatGPT's adeptness at contextual comprehension.

Motivation & Objective

• To evaluate the effectiveness of four generative AI chatbots—ChatGPT, Bing Chat, Bard, and Claude—as tools to enhance chemistry learning.
• To explore how these AI systems function as 'agents-to-think-with' by supporting critical thinking, problem-solving, and comprehension in a constructivist learning framework.
• To identify the role of prompt engineering in eliciting high-quality, nuanced responses from generative AI in educational contexts.
• To assess the implications for educator training and integration of AI tools into formal chemistry instruction.
• To determine which AI model offers the most effective support for personalized, inquiry-based chemistry learning.

Proposed method

• Conducted a single-case study using a simulated student persona engaged in chemistry learning tasks.
• Employed interaction logs between the AI models and the simulated learner to analyze discourse patterns and response quality.
• Applied content analysis methodology to systematically evaluate the depth, accuracy, and pedagogical value of AI-generated responses.
• Focused on Socratic questioning, contextual understanding, and response precision as key evaluation dimensions.
• Used iterative prompt refinement to assess how prompt crafting influences AI performance and response quality.
• Evaluated models based on their ability to stimulate reflection, foster creativity, and support problem-solving in chemistry education.

Experimental results

Research questions

• RQ1 How do ChatGPT, Bing Chat, Bard, and Claude perform as 'agents-to-think-with' in supporting chemistry learning tasks?
• RQ2 Which AI model demonstrates the most effective use of Socratic questioning to promote critical thinking and conceptual understanding?
• RQ3 To what extent does prompt engineering influence the quality and depth of responses from generative AI in educational contexts?
• RQ4 How do the models compare in contextual comprehension, precision, and adaptability to learner needs during simulated chemistry learning?
• RQ5 What are the implications of these findings for integrating generative AI into formal chemistry education and teacher training?

Key findings

• ChatGPT significantly outperformed Bing Chat in contextual comprehension, response depth, and pedagogical effectiveness during chemistry learning simulations.
• Bard and Claude delivered more in-depth, precise, and nuanced responses than Bing Chat, though not as consistently as ChatGPT.
• All four models demonstrated strong potential as 'agents-to-think-with,' enhancing critical thinking, problem-solving, and creativity through interactive dialogue.
• Socratic-like questioning was most effectively employed by ChatGPT, aligning closely with constructionist learning principles.
• Prompt crafting was identified as a pivotal factor in eliciting high-quality, reflective, and accurate responses from generative AI models.
• The study underscores the need for educator training to effectively integrate these AI tools into chemistry classrooms and maximize their learning potential.