[论文解读] Exploring AI-Generated Text in Student Writing: How Does AI Help?

David James Woo, Hengky Susanto+3 more|arXiv (Cornell University)|Mar 10, 2023

Text Readability and Simplification被引用 15

一句话总结

该研究分析AI生成文本如何影响英语作为外语(EFL)学生写作，使用23名香港中学生及回归/聚类分析来评估AI词语使用对写作质量的影响。

ABSTRACT

English as foreign language_EFL_students' use of text generated from artificial intelligence_AI_natural language generation_NLG_tools may improve their writing quality. However, it remains unclear to what extent AI-generated text in these students' writing might lead to higher-quality writing. We explored 23 Hong Kong secondary school students' attempts to write stories comprising their own words and AI-generated text. Human experts scored the stories for dimensions of content, language and organization. We analyzed the basic organization and structure and syntactic complexity of the stories' AI-generated text and performed multiple linear regression and cluster analyses. The results show the number of human words and the number of AI-generated words contribute significantly to scores. Besides, students can be grouped into competent and less competent writers who use more AI-generated text or less AI-generated text compared to their peers. Comparisons of clusters reveal some benefit of AI-generated text in improving the quality of both high-scoring students' and low-scoring students' writing. The findings can inform pedagogical strategies to use AI-generated text for EFL students' writing and to address digital divides. This study contributes designs of NLG tools and writing activities to implement AI-generated text in schools.

研究动机与目标

• 了解AI生成文本是否能提高EFL学生的写作质量。
• 量化人类词语与AI词语的平衡与写作分数的关系。
• 识别具有不同AI使用模式的作者群体（熟练型 vs. 较不熟练型）。
• 为课堂教学策略和自然语言生成工具的设计提供参考。

提出的方法

• 参与者：23名香港中学生，在写作中将自身词语与AI生成的文本相结合。
• 人工专家基于内容、语言和结构对故事进行评分。
• 对AI生成文本的基本结构和句法复杂度进行分析。
• 统计分析包括多元线性回归和聚类分析。
• 解释人类词语计数与AI词语计数如何与得分相关。

实验结果

研究问题

• RQ1 人类词语数量和AI生成词语数量是否显著地影响写作分数？
• RQ2 是否可以基于AI使用模式将学生分成不同的写作者类型？
• RQ3 AI生成文本是否在高分学生或低分学生中提升写作质量？

主要发现

• 人类词语数量和AI生成词语数量对分数有显著贡献。
• 可以将学生分为熟练的和不太熟练的作者，他们使用更多AI生成文本或更少AI生成文本。
• 聚类比较揭示在提升高分和低分学生写作质量方面，AI生成文本有一定的益处。