[Paper Review] Towards Minimal Supervision BERT-based Grammar Error Correction

Yiyuan Li, Antonios Anastasopoulos+1 more|arXiv (Cornell University)|Jan 10, 2020

Natural Language Processing Techniques8 references4 citations

TL;DR

This paper proposes a minimal-supervision BERT-based approach for grammatical error correction (GEC) that decouples error detection from correction. It uses BERT to generate corrections by masking tokens identified as erroneous (via sequence labeling), achieving over 70% precision in single-edit settings, with potential for improvement through reranking and better span detection.

ABSTRACT

Current grammatical error correction (GEC) models typically consider the task as sequence generation, which requires large amounts of annotated data and limit the applications in data-limited settings. We try to incorporate contextual information from pre-trained language model to leverage annotation and benefit multilingual scenarios. Results show strong potential of Bidirectional Encoder Representations from Transformers (BERT) in grammatical error correction task.

Motivation & Objective

• To reduce reliance on large annotated datasets for grammatical error correction by leveraging pre-trained BERT models.
• To explore the feasibility of using BERT for GEC with minimal supervision, especially in low-resource or multilingual settings.
• To evaluate the performance of BERT in generating corrections when guided by oracle error spans, focusing on precision and recall in single-edit scenarios.
• To investigate the impact of different masking strategies and reranking mechanisms on correction quality.
• To identify key failure modes such as redundant edits and hallucinations in BERT-generated corrections.

Proposed method

• The GEC task is split into two stages: error identification via sequence labeling (labels: remain, substitution, insert, delete), followed by correction using BERT.
• Masking is applied to tokens marked for substitution or insertion, with BERT generating candidate corrections for each masked span.
• Different masking strategies are evaluated: masking based on original edit span length, final correction length, or using a single mask per sentence.
• Subword predictions (e.g., 'ad', '##e', '##quate') are merged at the sentence level during evaluation to assess full sentence correctness.
• A reranking mechanism is explored to improve performance by selecting the best correction from top-k BERT predictions.
• Future work includes modeling error fertility to predict the number of [MASK] tokens and integrating fluency and grammar-aware scoring in reranking.

Experimental results

Research questions

• RQ1 Can BERT generate accurate grammatical corrections when guided by minimal supervision and oracle error spans?
• RQ2 How do different masking strategies (span-based, correction-based, single-mask) affect correction performance?
• RQ3 To what extent can reranking improve the selection of correct corrections from BERT’s top-k predictions?
• RQ4 What are the main failure modes of BERT in GEC, such as redundant edits or hallucinations?
• RQ5 How can error span detection and partial masking strategies be improved to reduce over-generativity in BERT outputs?

Key findings

• Using a single mask per sentence yields the highest performance, achieving 79.0% F0.5 score on the 'each edit' split and 79.4% on the 'last edit' split.
• BERT achieves over 70% precision (P@1) in correction generation, with 76.3% and 76.7% P@1 on the 'each edit' and 'last edit' splits, respectively.
• Performance improves significantly when considering top-5 candidates, with Acc@5 reaching 55.4% on the 'each edit' split, indicating that reranking can enhance results.
• Common failure cases include redundant edits (e.g., adding 'stop' where a comma is needed) and hallucinations (e.g., replacing 'number 8 bus' with 'small parking station').
• The model shows strong potential for minimal-supervision GEC, especially when combined with reranking and improved error span detection.