[Paper Review] Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods

Mengsay Loem, Masahiro Kaneko+2 more|arXiv (Cornell University)|May 29, 2023

Natural Language Processing Techniques4 citations

TL;DR

This study evaluates GPT-3's performance and controllability in Grammatical Error Correction (GEC) using prompt-based zero-shot and few-shot methods. It demonstrates that GPT-3 achieves competitive GEC performance and can be effectively controlled via natural language instructions and examples, particularly for minimal edits, fluency improvements, and learner-level-specific corrections.

ABSTRACT

Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks. However, applying prompt-based methods with GPT-3 for Grammatical Error Correction (GEC) tasks and their controllability remains underexplored. Controllability in GEC is crucial for real-world applications, particularly in educational settings, where the ability to tailor feedback according to learner levels and specific error types can significantly enhance the learning process. This paper investigates the performance and controllability of prompt-based methods with GPT-3 for GEC tasks using zero-shot and few-shot setting. We explore the impact of task instructions and examples on GPT-3's output, focusing on controlling aspects such as minimal edits, fluency edits, and learner levels. Our findings demonstrate that GPT-3 could effectively perform GEC tasks, outperforming existing supervised and unsupervised approaches. We also showed that GPT-3 could achieve controllability when appropriate task instructions and examples are given.

Motivation & Objective

• To evaluate the performance of GPT-3 in Grammatical Error Correction (GEC) using prompt-based zero-shot and few-shot settings.
• To investigate whether GPT-3 can be effectively controlled to produce specific types of corrections, such as minimal edits or fluency-focused revisions.
• To examine the impact of task instructions and example selection on GPT-3’s GEC output quality and controllability.
• To assess the scalability and adaptability of prompt-based GEC methods in educational contexts without requiring large-scale labeled datasets.
• To provide insights into designing effective prompts for personalized language feedback in intelligent tutoring systems.

Proposed method

• Utilized GPT-3 (text-davinci-003) via OpenAI API for GEC tasks in both zero-shot and few-shot settings.
• Designed natural language prompts with varying task instructions to guide GPT-3 on correction style, e.g., 'Correct the grammatical errors in the following sentence'.
• Implemented in-context learning in few-shot settings by including 1–5 labeled examples before the input text.
• Varied the phrasing of task instructions and selected diverse, representative examples to assess their impact on output quality.
• Evaluated model outputs using standard GEC metrics such as edit distance and task-specific scores on JFLEG, CoNLL2014, and W&I benchmarks.
• Conducted ablation studies to isolate the effects of instructions and examples on controllability and performance.

Experimental results

Research questions

• RQ1 To what extent can GPT-3 perform GEC tasks using only prompt-based methods in zero-shot and few-shot settings?
• RQ2 Can task instructions alone control GPT-3’s correction style, such as focusing on minimal edits or fluency?
• RQ3 How does the number of few-shot examples affect GPT-3’s GEC performance and controllability?
• RQ4 Can GPT-3 be effectively guided to produce corrections tailored to specific learner proficiency levels?
• RQ5 What is the relative contribution of task instructions versus example examples in achieving controllable GEC outputs?

Key findings

• GPT-3 achieved competitive GEC performance in both zero-shot and few-shot settings, outperforming existing supervised and unsupervised methods on standard benchmarks.
• Task instructions alone were sufficient to control GPT-3’s correction style, such as prioritizing minimal edits or fluency improvements.
• The inclusion of few-shot examples significantly improved performance, with gains increasing non-linearly as the number of examples rose.
• Combining well-crafted task instructions with high-quality examples led to the most effective and controllable GEC outputs.
• GPT-3 demonstrated strong adaptability to learner-level-specific corrections, suggesting potential for personalized language feedback in educational tools.
• The study suggests that prompt engineering with GPT-3 can reduce reliance on large-scale annotated datasets for GEC, enabling scalable deployment in diverse learning environments.