[Paper Review] Exploring the Impact of ChatGPT on Student Interactions in Computer-Supported Collaborative Learning

Han Kyul Kim, Shriniwas Nayak+4 more|arXiv (Cornell University)|Mar 11, 2024

Artificial Intelligence in Healthcare and Education6 citations

TL;DR

The study investigates how incorporating ChatGPT as an instantaneous Q&A agent in asynchronous CSCL brainstorming alters student interactions, finding no overall change in active learning but significant shifts: higher active learning in student–ChatGPT interactions and a 22.6 percentage point increase in active learning within student–student interactions in Type B cohorts.

ABSTRACT

The growing popularity of generative AI, particularly ChatGPT, has sparked both enthusiasm and caution among practitioners and researchers in education. To effectively harness the full potential of ChatGPT in educational contexts, it is crucial to analyze its impact and suitability for different educational purposes. This paper takes an initial step in exploring the applicability of ChatGPT in a computer-supported collaborative learning (CSCL) environment. Using statistical analysis, we validate the shifts in student interactions during an asynchronous group brainstorming session by introducing ChatGPT as an instantaneous question-answering agent.

Motivation & Objective

• Explore applicability of ChatGPT in CSCL for creative thinking development in a graduate product engineering setting.
• Quantitatively analyze changes in student interaction patterns when ChatGPT is available.
• Assess whether ChatGPT affects overall active learning and inter-student dynamics in asynchronous group brainstorming.

Proposed method

• Graduate CSCL dataset from a 2023 product engineering course with 12 cohorts (6 Type A with ChatGPT, 6 Type B without).
• ChatGPT responses are posted to tagged messages via OpenAI API, enabling instantaneous QA within groups.
• Active learning messages are annotated using the Collaborative Learning Conversation Skills Taxonomy (CLCST) focusing on Request, Inform, and Motivate.
• Inter-annotator reliability measured with Cohen’s kappa = 0.932 (almost perfect agreement).
• Repeated ANOVA with ChatGPT presence and week as covariates; Tukey post hoc for pairwise comparisons.

Experimental results

Research questions

• RQ1 Does introducing ChatGPT change the overall level of active learning in CSCL group discussions?
• RQ2 How does ChatGPT presence affect the balance between student–student and student–ChatGPT interactions?
• RQ3 Are there interaction-type–specific effects (student–student vs. student–ChatGPT) on active learning across weeks?

Key findings

• There is no statistically significant difference in overall active learning between Type A (with ChatGPT) and Type B (without ChatGPT) cohorts.
• Type B cohorts show a 22.6 percentage point higher ratio of active learning in student–student interactions compared to Type A cohorts (p-adjusted = 0.037).
• Within Type A cohorts, student–ChatGPT interactions exhibit a significantly higher level of active learning than student–student interactions, with an average difference of 53.57 percentage points (p-adjusted = 0.000).
• ANOVA for student–student interactions shows ChatGPT as a significant factor (F = 4.909, p < 0.05) in altering active-learning ratios.
• Week and interaction terms show limited or no significant effects on the measured outcomes.
• Conclusion notes that ChatGPT reduces overall student-student active learning in aggregate but increases bilateral activity between students and ChatGPT; novelty effects were not explicitly accounted for.