[Paper Review] Rethinking the Roles of Large Language Models in Chinese Grammatical Error Correction

Yinghui Li, Shang Qin+7 more|arXiv (Cornell University)|Feb 18, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes EXAM and SEE, frameworks that reposition large language models (LLMs) as explainers and evaluators in Chinese Grammatical Error Correction (CGEC), respectively. By leveraging LLMs' grammatical knowledge and semantic understanding to enhance small models during training and improve evaluation consistency, the approach significantly boosts small-model performance and aligns evaluation more closely with human judgment, demonstrating superior results on benchmark datasets.

ABSTRACT

Recently, Large Language Models (LLMs) have been widely studied by researchers for their roles in various downstream NLP tasks. As a fundamental task in the NLP field, Chinese Grammatical Error Correction (CGEC) aims to correct all potential grammatical errors in the input sentences. Previous studies have shown that LLMs' performance as correctors on CGEC remains unsatisfactory due to its challenging task focus. To promote the CGEC field to better adapt to the era of LLMs, we rethink the roles of LLMs in the CGEC task so that they can be better utilized and explored in CGEC. Considering the rich grammatical knowledge stored in LLMs and their powerful semantic understanding capabilities, we utilize LLMs as explainers to provide explanation information for the CGEC small models during error correction to enhance performance. We also use LLMs as evaluators to bring more reasonable CGEC evaluations, thus alleviating the troubles caused by the subjectivity of the CGEC task. In particular, our work is also an active exploration of how LLMs and small models better collaborate in downstream tasks. Extensive experiments and detailed analyses on widely used datasets verify the effectiveness of our thinking intuition and the proposed methods.

Motivation & Objective

• To address the poor performance of LLMs as direct correctors in Chinese Grammatical Error Correction (CGEC) due to violation of the minimum change principle.
• To explore how LLMs can be effectively repurposed beyond direct correction, leveraging their strengths in explanation and evaluation.
• To improve the training of small CGEC models by incorporating high-quality, LLM-generated explanations of errors and corrections.
• To develop a more human-aligned evaluation framework for CGEC that reduces subjectivity by using LLMs as evaluators.
• To enable collaborative synergy between LLMs and small models in downstream NLP tasks, particularly in low-resource or high-precision scenarios like CGEC.

Proposed method

• EXAM uses LLMs to generate detailed explanations—covering error types, reference corrections, and linguistic justifications—for grammatically incorrect sentences.
• These LLM-generated explanations are distilled into a fine-tuned small CGEC model during training to improve its correction accuracy.
• SEE employs LLMs as evaluators that assess model outputs based on both grammatical correctness and semantic preservation, ensuring alignment with human judgment.
• The evaluation process in SEE balances edits in golden-standard annotations with model-generated edits, promoting fairness and comprehensiveness.
• Both frameworks are evaluated on widely used CGEC benchmarks, with ablation studies confirming the contribution of each component.
• The method relies on prompting strategies to extract structured, high-quality explanation and evaluation signals from LLMs like GPT-3.5-Turbo and Qwen-72B-Chat.

Experimental results

Research questions

• RQ1 Can LLMs serve as effective explainers to improve the performance of small CGEC models through knowledge distillation?
• RQ2 How can LLMs be leveraged to create more human-aligned and comprehensive evaluation metrics for CGEC?
• RQ3 Does using LLMs as evaluators reduce the subjectivity inherent in traditional automatic evaluation of GEC?
• RQ4 To what extent does LLM-generated explanation improve the generalization and robustness of small CGEC models?
• RQ5 Can the collaboration between LLMs and small models lead to better overall performance than direct LLM fine-tuning on CGEC?

Key findings

• EXAM significantly improves the performance of small CGEC models by incorporating LLM-generated explanations, outperforming direct fine-tuning of LLMs on the same task.
• SEE evaluation results are consistently closer to human judgment than traditional metrics like BLEU and F0.5, especially for small models.
• SEE evaluation for LLMs slightly surpasses human evaluation, indicating that LLMs can achieve more consistent and nuanced assessment than humans in this context.
• Human evaluation confirms that SEE's judgments align more closely with human annotators than standard automatic metrics.
• Case studies show that LLMs provide high-quality, consistent, and actionable explanations for grammatical errors, even when error types vary.
• The framework demonstrates that LLMs can be effectively repurposed as explainers and evaluators, bypassing the limitations of direct correction due to the minimum change principle.