[Paper Review] Evaluating GPT-3.5 and GPT-4 on Grammatical Error Correction for Brazilian Portuguese

Maria Carolina Penteado, Fábio Perez|arXiv (Cornell University)|Jun 27, 2023

Natural Language Processing Techniques4 citations

TL;DR

This study evaluates GPT-3.5 and GPT-4 as Grammatical Error Correction (GEC) tools for Brazilian Portuguese, introducing a new four-category dataset (Grammar, Spelling, Fast typing, Internet language). GPT-4 outperforms GPT-3.5 and traditional tools like Microsoft Word and Google Docs, especially in non-traditional error types, though all LLMs exhibit overcorrection due to low precision despite high recall.

ABSTRACT

We investigate the effectiveness of GPT-3.5 and GPT-4, two large language models, as Grammatical Error Correction (GEC) tools for Brazilian Portuguese and compare their performance against Microsoft Word and Google Docs. We introduce a GEC dataset for Brazilian Portuguese with four categories: Grammar, Spelling, Internet, and Fast typing. Our results show that while GPT-4 has higher recall than other methods, LLMs tend to have lower precision, leading to overcorrection. This study demonstrates the potential of LLMs as practical GEC tools for Brazilian Portuguese and encourages further exploration of LLMs for non-English languages and other educational settings.

Motivation & Objective

• To assess the effectiveness of GPT-3.5 and GPT-4 as Grammatical Error Correction (GEC) tools for Brazilian Portuguese.
• To compare LLMs against traditional tools like Microsoft Word and Google Docs in correcting various error types.
• To develop and release a new, annotated GEC dataset for Brazilian Portuguese covering Grammar, Spelling, Fast typing, and Internet language errors.
• To investigate the limitations of LLMs in GEC, particularly overcorrection and precision issues, in non-English linguistic contexts.

Proposed method

• Created a new GEC dataset with 484 sentences divided into four categories: Grammar (102 pairs), Spelling (100), Fast typing (40), and Internet language (40).
• Designed a zero-shot prompting strategy where models were prompted to correct input sentences and return only the corrected version.
• Applied automatic evaluation using F0.5 score, precision, recall, and F1, with human evaluation for qualitative analysis.
• Conducted qualitative analysis to classify correction behaviors: over-correction, omission, grammatical miscorrection, and ungrammatical miscorrection.
• Evaluated models on four error categories to assess performance across different linguistic complexities.
• Used both automatic metrics and manual inspection to identify systematic errors and biases in LLM outputs.

Experimental results

Research questions

• RQ1 How do GPT-3.5 and GPT-4 perform compared to Microsoft Word and Google Docs in correcting grammatical errors in Brazilian Portuguese?
• RQ2 To what extent do LLMs overcorrect or undercorrect in different error categories, particularly in informal or context-dependent language?
• RQ3 What are the main failure modes of LLMs in GEC for Brazilian Portuguese, such as over-correction, meaning change, or ungrammatical corrections?
• RQ4 How does the performance of GPT-4 compare to GPT-3.5 in terms of precision, recall, and F0.5 score across error categories?
• RQ5 Can LLMs effectively handle non-standard language forms like internet slang and fast-typing errors, where rule-based systems often fail?

Key findings

• GPT-4 achieved the highest F0.5 score in both Grammar and Spelling categories, outperforming GPT-3.5, Microsoft Word, and Google Docs.
• GPT-4 demonstrated higher recall than other methods, indicating better error detection, but suffered from lower precision due to overcorrection.
• GPT-3.5 produced 54 over-corrections, while GPT-4 produced only six, showing a significant reduction in unnecessary edits.
• In the Internet and Fast typing categories, GPT-3.5 and GPT-4 significantly outperformed Microsoft Word and Google Docs, which struggled with informal and context-dependent errors.
• GPT-4 had 13 grammatical miscorrections and 1 ungrammatical miscorrection, compared to 41 and 3 for GPT-3.5, respectively, indicating better overall accuracy.
• Despite high recall, LLMs showed a tendency to alter correct sentences unnecessarily, highlighting a key limitation in precision for real-world GEC applications.