[Paper Review] ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

Haoran Wu, Wenxuan Wang+3 more|arXiv (Cornell University)|Mar 15, 2023

Text Readability and SimplificationComputer Science48 citations

TL;DR

The paper evaluates ChatGPT on the Grammatical Error Correction task using CoNLL-2014, comparing it with Grammarly and GECToR, and analyzes automatic vs. human evaluations to reveal strengths in surface-editing and weaknesses in long-sentence automatic metrics.

ABSTRACT

ChatGPT is a cutting-edge artificial intelligence language model developed by OpenAI, which has attracted a lot of attention due to its surprisingly strong ability in answering follow-up questions. In this report, we aim to evaluate ChatGPT on the Grammatical Error Correction(GEC) task, and compare it with commercial GEC product (e.g., Grammarly) and state-of-the-art models (e.g., GECToR). By testing on the CoNLL2014 benchmark dataset, we find that ChatGPT performs not as well as those baselines in terms of the automatic evaluation metrics (e.g., $F_{0.5}$ score), particularly on long sentences. We inspect the outputs and find that ChatGPT goes beyond one-by-one corrections. Specifically, it prefers to change the surface expression of certain phrases or sentence structure while maintaining grammatical correctness. Human evaluation quantitatively confirms this and suggests that ChatGPT produces less under-correction or mis-correction issues but more over-corrections. These results demonstrate that ChatGPT is severely under-estimated by the automatic evaluation metrics and could be a promising tool for GEC.

Motivation & Objective

• Assess ChatGPT's effectiveness for grammatical error correction (GEC).
• Compare ChatGPT against Grammarly and a state-of-the-art GEC model (GECToR) on CoNLL-2014.
• Analyze how sentence length affects GEC performance and examine error types and human judgments.
• Demonstrate limitations of automatic metrics and explore potential of ChatGPT for GEC.

Proposed method

• Evaluate on CoNLL-2014 test subset (100 sentences) using a fixed prompt for ChatGPT.
• Compare ChatGPT with Grammarly and GECToR using Precision, Recall, and F0.5 metrics.
• Use the CoNLL-2014 official scorer (adapted to current Python) for automatic evaluation.
• Perform qualitative analysis of outputs, including example corrections and error-type categories.
• Conduct a small human evaluation (20 sentences) to classify under-correction, mis-correction, and over-correction.
• Analyze long vs. short sentence performance and the effect of post-hoc corrections (Grammarly on outputs).

Experimental results

Research questions

• RQ1 Is ChatGPT a good tool for grammatical error correction (GEC) compared to Grammarly and GECToR?
• RQ2 How does ChatGPT perform on the CoNLL-2014 GEC benchmark across different sentence lengths?
• RQ3 Do automatic evaluation metrics align with human judgments for ChatGPT in GEC?
• RQ4 What are the qualitative characteristics of ChatGPT's corrections (e.g., one-by-one corrections vs surface/structural edits)?

Key findings

• ChatGPT achieves high recall but lower precision and an overall F0.5 of 53.1, underperforming compared to Grammarly (63.3) and GECToR (60.8) on CoNLL-2014.
• ChatGPT tends to correct more errors (higher recall) but also introduces more over-corrections (low precision).
• GECToR has the highest precision, Grammarly provides a balanced performance, while ChatGPT emphasizes broader edits beyond one-by-one corrections.
• Long sentences reveal a notable drop in ChatGPT's F0.5 performance relative to Grammarly and GECToR.
• Human evaluation shows ChatGPT has the fewest under-corrections (3) and mis-corrections (3) but the most over-corrections (30) among the three systems.
• Grammarly can provide marginal improvements to ChatGPT outputs, particularly in punctuation, but has limited impact on some GEC errors.