[Paper Review] Better by you, better than me, chatgpt3 as writing assistance in students essays

Željana Bašić, Ana Banovac+2 more|arXiv (Cornell University)|Feb 9, 2023

Artificial Intelligence in Healthcare and EducationMedicine16 references36 citations

TL;DR

The study compares student essay quality when using ChatGPT-3 as a writing assistant versus not using it; it finds no improvement with the tool and even hints at lower performance in the experimental group.

ABSTRACT

Aim: To compare students' essay writing performance with or without employing ChatGPT-3 as a writing assistant tool. Materials and methods: Eighteen students participated in the study (nine in control and nine in the experimental group that used ChatGPT-3). We scored essay elements with grades (A-D) and corresponding numerical values (4-1). We compared essay scores to students' GPTs, writing time, authenticity, and content similarity. Results: Average grade was C for both groups; for control (2.39, SD=0.71) and for experimental (2.00, SD=0.73). None of the predictors affected essay scores: group (P=0.184), writing duration (P=0.669), module (P=0.388), and GPA (P=0.532). The text unauthenticity was slightly higher in the experimental group (11.87%, SD=13.45 to 9.96%, SD=9.81%), but the similarity among essays was generally low in the overall sample (the Jaccard similarity index ranging from 0 to 0.054). In the experimental group, AI classifier recognized more potential AI-generated texts. Conclusions: This study found no evidence that using GPT as a writing tool improves essay quality since the control group outperformed the experimental group in most parameters.

Motivation & Objective

• Assess whether ChatGPT-3 as a writing assistant improves student essay quality.
• Compare writing time, authenticity, and content similarity between groups with and without AI assistance.
• Evaluate AI-generated text detectability and its relation to essay scores.

Proposed method

• Random assignment of 18 students to control (no ChatGPT-3) or experimental (with ChatGPT-3) groups.
• Essay scoring using A-D grades mapped to numerical values (4-1).
• Comparison of essay scores, writing duration, authenticity, and content similarity between groups.
• Calculation of Jaccard similarity index for content similarity across essays.
• AI-text classifier used to assess potential AI-generated text in the experimental group.

Experimental results

Research questions

• RQ1 Does using ChatGPT-3 as a writing assistant improve overall essay grades?
• RQ2 How does writing duration affect essay quality with or without AI assistance?
• RQ3 Does AI-assisted writing affect authenticity and content similarity among students’ essays?
• RQ4 Can AI-detection tools reliably recognize AI-assisted writing in student essays?

Key findings

• Average grade was C for both groups (control 2.39, SD=0.71; experimental 2.00, SD=0.73).
• Group, writing duration, module, and GPA did not significantly affect essay scores (P-values: 0.184, 0.669, 0.388, 0.532 respectively).
• Text unauthenticity was slightly higher in the experimental group (11.87%, SD=13.45) than in the control group (9.96%, SD=9.81%).
• Jaccard similarity index indicated generally low content similarity across the overall sample (0 to 0.054).
• In the experimental group, the AI classifier identified more potential AI-generated texts.