[Paper Review] Controlled Generation with Prompt Insertion for Natural Language Explanations in Grammatical Error Correction

Masahiro Kaneko, Naoaki Okazaki|arXiv (Cornell University)|Sep 20, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

This paper proposes a controlled generation method using Prompt Insertion (PI) to guide large language models (LLMs) in generating comprehensive, natural language explanations for grammatical error corrections in GEC. By extracting error-correction pairs and inserting them as dynamic prompts during inference, PI ensures all corrections are explicitly addressed, significantly improving explanation coverage and quality over standard prompting, as validated on a new XGEC dataset and across GPT-3.5 and ChatGPT models.

ABSTRACT

In Grammatical Error Correction (GEC), it is crucial to ensure the user's comprehension of a reason for correction. Existing studies present tokens, examples, and hints as to the basis for correction but do not directly explain the reasons for corrections. Although methods that use Large Language Models (LLMs) to provide direct explanations in natural language have been proposed for various tasks, no such method exists for GEC. Generating explanations for GEC corrections involves aligning input and output tokens, identifying correction points, and presenting corresponding explanations consistently. However, it is not straightforward to specify a complex format to generate explanations, because explicit control of generation is difficult with prompts. This study introduces a method called controlled generation with Prompt Insertion (PI) so that LLMs can explain the reasons for corrections in natural language. In PI, LLMs first correct the input text, and then we automatically extract the correction points based on the rules. The extracted correction points are sequentially inserted into the LLM's explanation output as prompts, guiding the LLMs to generate explanations for the correction points. We also create an Explainable GEC (XGEC) dataset of correction reasons by annotating NUCLE, CoNLL2013, and CoNLL2014. Although generations from GPT-3 and ChatGPT using original prompts miss some correction points, the generation control using PI can explicitly guide to describe explanations for all correction points, contributing to improved performance in generating correction reasons.

Motivation & Objective

• To address the lack of direct, natural language explanations for grammatical error corrections in existing GEC systems.
• To improve the comprehensibility and reliability of LLM-generated explanations by explicitly controlling which corrections are explained.
• To create a new dataset, XGEC, for training and evaluating explainable GEC systems with human-annotated correction reasons.
• To investigate whether LLM-generated explanations can match human-level quality in improving downstream GEC performance.

Proposed method

• First, the LLM performs standard grammatical error correction on the input text.
• Then, token-level alignment between input and corrected output is computed using rule-based methods to extract error-correction pairs.
• These extracted correction pairs are inserted as structured prompts during the explanation generation phase to guide the LLM.
• The LLM is prompted to generate a natural language explanation for each inserted correction pair, ensuring all edits are addressed.
• The method supports both post-correction and pre-correction explanation generation, with ablation showing post-correction is more effective.
• A new Explainable GEC (XGEC) dataset is constructed by annotating correction reasons on NUCLE, CoNLL2013, and CoNLL2014.

Figure 1: How to generate an explanation of the proposed method PI.

Experimental results

Research questions

• RQ1 Can LLMs generate comprehensive, natural language explanations for grammatical error corrections without explicit control?
• RQ2 How does prompt insertion of correction pairs affect the coverage and quality of LLM-generated explanations?
• RQ3 Does the quality of LLM-generated explanations match that of human-annotated explanations in improving downstream GEC performance?
• RQ4 Is post-correction explanation generation more effective than pre-correction generation in terms of edit coverage?

Key findings

• PI significantly improves explanation coverage: human evaluation shows 2.0 coverage score (perfect coverage) for both GPT-3.5 and ChatGPT with PI, compared to lower scores without PI.
• PI enhances explanation quality: GPT-3.5 and ChatGPT with PI achieve 1.8 and 1.9 validity scores respectively, indicating high accuracy and usefulness for language learners.
• Automated evaluation shows PI improves F1 scores by up to 12.5% on XGECa and 10.3% on XGECb compared to standard prompting.
• LLM-generated explanations using PI achieve GEC performance comparable to human-annotated explanations in few-shot learning, with minimal performance difference on CoNLL2014, W&I, and JFLEG datasets.
• Post-correction explanation generation outperforms pre-correction generation in edit coverage, with a 15% higher coverage score in human evaluation.
• The method effectively mitigates omission and ambiguity issues present in standard prompt-based generation, where models often miss or conflate correction points.