[Paper Review] ChatGPT for Arabic Grammatical Error Correction

Sang Yun Kwon, Gagan Bhatia+2 more|arXiv (Cornell University)|Aug 8, 2023

Topic Modeling4 citations

TL;DR

This paper investigates the effectiveness of instruction-fine-tuned large language models (LLMs), particularly ChatGPT, for Arabic Grammatical Error Correction (AGEC), a low-resource, morphologically complex task. It demonstrates that few-shot prompting and synthetic data generation significantly boost performance, achieving new state-of-the-art F1 scores of 72.19 and 73.26 on the QALB-2014 and QALB-2015 benchmarks, respectively, outperforming both baseline models and fully fine-tuned architectures.

ABSTRACT

Recently, large language models (LLMs) fine-tuned to follow human instruction have exhibited significant capabilities in various English NLP tasks. However, their performance in grammatical error correction (GEC) tasks, particularly in non-English languages, remains significantly unexplored. In this paper, we delve into abilities of instruction fine-tuned LLMs in Arabic GEC, a task made complex due to Arabic's rich morphology. Our findings suggest that various prompting methods, coupled with (in-context) few-shot learning, demonstrate considerable effectiveness, with GPT-4 achieving up to $65.49$ F extsubscript{1} score under expert prompting (approximately $5$ points higher than our established baseline). This highlights the potential of LLMs in low-resource settings, offering a viable approach for generating useful synthetic data for model training. Despite these positive results, we find that instruction fine-tuned models, regardless of their size, significantly underperform compared to fully fine-tuned models of significantly smaller sizes. This disparity highlights a substantial room for improvements for LLMs. Inspired by methods from low-resource machine translation, we also develop a method exploiting synthetic data that significantly outperforms previous models on two standard Arabic benchmarks. Our work sets new SoTA for Arabic GEC, with $72.19\%$ and $73.26$ F$_{1}$ on the 2014 and 2015 QALB datasets, respectively.

Motivation & Objective

• To evaluate the performance of instruction-fine-tuned LLMs like ChatGPT on Arabic GEC, a low-resource, morphologically rich language task.
• To investigate the impact of different prompting strategies—such as few-shot chain-of-thought and expert prompting—on GEC performance.
• To assess the effectiveness of synthetic data generated by LLMs in improving downstream model performance on Arabic GEC.
• To compare the performance of LLM-based approaches with traditional sequence-to-sequence and sequence-to-edit models on standard Arabic benchmarks.
• To analyze error types and identify key challenges in Arabic GEC, particularly related to morphology and low-resource data scarcity.

Proposed method

• The study employs few-shot prompting with chain-of-thought (CoT) and expert prompting (EP) strategies to guide ChatGPT in correcting Arabic grammatical errors.
• It generates synthetic training data using ChatGPT, applying careful filtering to ensure high-quality, in-domain examples for fine-tuning subsequent models.
• A sequence-to-sequence (seq2seq) model is trained on both real and synthetic data to evaluate the impact of data augmentation on GEC performance.
• A sequence-to-edit (seq2edit) model is implemented to compare correction strategies, focusing on edit operations rather than full sequence generation.
• The performance of LLMs is evaluated using standard metrics (precision, recall, F1) on the QALB-2014 and QALB-2015 benchmarks under various prompting and data settings.
• Error analysis is conducted using the ARETA framework to categorize and quantify error types, such as character replacement, missing words, and hamza errors.

Figure 1: An example of an Arabic GEC system showcasing six types of errors: character replacement , missing word , hamza error , missing punctuation , additional character , and punctuation confusion .

Experimental results

Research questions

• RQ1 Can instruction-fine-tuned LLMs like ChatGPT achieve competitive performance in Arabic GEC with minimal in-context examples?
• RQ2 How does the quality and domain alignment of synthetic data generated by LLMs affect downstream model performance in low-resource AGEC settings?
• RQ3 What is the relative effectiveness of different prompting strategies—such as few-shot CoT and expert prompting—for Arabic GEC?
• RQ4 How do LLM-based approaches compare to fully fine-tuned seq2seq and seq2edit models in terms of F1 score and error detection capabilities?
• RQ5 What trade-offs exist between precision and recall when scaling synthetic training data in Arabic GEC?

Key findings

• GPT-4 achieved an F1 score of 65.49 under expert prompting, approximately 5 points higher than the established baseline, demonstrating strong few-shot generalization in Arabic GEC.
• The proposed method using synthetic data generated by ChatGPT significantly outperformed previous models, achieving new state-of-the-art F1 scores of 72.19 on QALB-2014 and 73.26 on QALB-2015.
• Despite strong few-shot performance, instruction-fine-tuned LLMs like ChatGPT underperformed compared to smaller, fully fine-tuned models, indicating a significant gap in capability for this task.
• Data augmentation showed a trade-off between precision and recall: as dataset size increased, precision improved but recall decreased, highlighting a challenge in scaling synthetic data.
• Models trained with out-of-domain synthetic data underperformed, confirming that data quality is as critical as quantity in low-resource NLP tasks.
• The sequence-to-tagging approach showed high precision but low recall, suggesting a limitation in error detection due to the absence of effective G-transformations for Arabic’s rich morphology.

Figure 2: An illustration of Few-Shot CoT and Expert Prompts applied to CharGPT for Grammatical Error Correction.