[Paper Review] LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction

Jieun Han, Haneul Yoo+9 more|arXiv (Cornell University)|Oct 8, 2023

Online Learning and Analytics4 citations

TL;DR

This paper introduces FABRIC, a pipeline that enhances EFL writing education by generating rubric-based scores (content, organization, language) and detailed, helpful feedback using LLMs. It combines a new real-world dataset (DREsS), a corruption-based data augmentation strategy (CASE), and a novel Essay Chain-of-Thought prompting method (EssayCoT), achieving significant improvements in feedback helpfulness and model performance over baselines.

ABSTRACT

In the context of English as a Foreign Language (EFL) writing education, LLM-as-a-tutor can assist students by providing real-time feedback on their essays. However, challenges arise in assessing LLM-as-a-tutor due to differing standards between educational and general use cases. To bridge this gap, we integrate pedagogical principles to assess student-LLM interaction. First, we explore how LLMs can function as English tutors, providing effective essay feedback tailored to students. Second, we propose three metrics to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects. In this process, EFL experts evaluate the feedback from LLM-as-a-tutor regarding quality and characteristics. On the other hand, EFL learners assess their learning outcomes from interaction with LLM-as-a-tutor. This approach lays the groundwork for developing LLMs-as-a-tutor tailored to the needs of EFL learners, advancing the effectiveness of writing education in this context.

Motivation & Objective

• To address the lack of standardized, rubric-based datasets for automated essay scoring (AES) in EFL writing education.
• To improve the accuracy of rubric-based AES models through a novel data augmentation strategy (CASE) that introduces controlled errors.
• To develop a prompting method (EssayCoT) that generates more helpful, specific feedback by leveraging predicted rubric scores.
• To evaluate the end-to-end pipeline in real-world EFL writing classrooms, assessing both model performance and user satisfaction.

Proposed method

• FABRIC is a three-component pipeline: DREsS (a real-world dataset of 1,782 EFL essays scored by experts on content, organization, and language), CASE (corruption-based data augmentation generating synthetic essays with errors to improve model generalization), and EssayCoT (a prompting strategy that uses predicted rubric scores to guide LLM feedback generation).
• DREsS is constructed from actual EFL student essays and includes expert-annotated rubric-based scores, forming a standardized benchmark for rubric-based AES.
• CASE generates synthetic training data by introducing plausible sentence-level errors into high-quality essays, creating 3.9K content, 15.7K organization, and 0.9K language synthetic samples to enhance model robustness.
• EssayCoT uses the predicted rubric scores from the AES model as a chain-of-thought to guide the LLM in generating feedback that is more targeted and coherent than standard prompting.
• The AES model is fine-tuned on DREsS and augmented with CASE, improving its quadratic weighted kappa (QWK) score by 26.37% over the baseline.
• Feedback is evaluated via expert assessment and student deployment, with the pipeline integrated into an essay editing platform for real-world testing.

Figure 1: Overview of the pipeline. Rubric-based AES data (DREsS) is used to train AES model, which is enhanced by CASE to more accurately predict rubric-based scores. EssayCoT leverages these scores for essay feedback generation. FABRIC’s final outputs, scores and feedback, are used for EFL writing

Experimental results

Research questions

• RQ1 Can a standardized, expert-annotated rubric-based dataset (DREsS) improve the reliability and consistency of automated essay scoring in EFL writing education?
• RQ2 Does corruption-based data augmentation (CASE) significantly enhance the performance of rubric-based AES models compared to baseline training on existing datasets?
• RQ3 Can a chain-of-thought prompting strategy (EssayCoT) that uses predicted rubric scores generate feedback that is significantly more helpful and preferred by English education experts than standard prompting?
• RQ4 How do students perceive and rate AI-generated scores and feedback in a real-world EFL writing classroom setting?

Key findings

• The DREsS dataset, comprising 1,782 real EFL student essays with expert-annotated scores on content, organization, and language, provides a reliable benchmark for rubric-based AES.
• The CASE augmentation strategy improved the baseline model’s quadratic weighted kappa (QWK) score by 26.37%, significantly enhancing the accuracy of rubric-based scoring.
• Feedback generated using EssayCoT prompting was rated as significantly more helpful and preferred by 13 English education experts compared to standard prompting methods.
• In a real-world deployment with 33 EFL students, the generated scores and feedback received an average Likert rating of 6 out of 7, indicating strong user acceptance.
• The pipeline achieved a QWK score of approximately 0.6 on the augmented dataset, demonstrating strong performance in rubric-based scoring.
• Expert evaluation confirmed that EssayCoT-generated feedback was more specific, actionable, and aligned with pedagogical goals than feedback from standard prompting.

Figure 2: Prompt for EssayCoT