[Paper Review] Chinese Grammatical Correction Using BERT-based Pre-trained Model

Hongfei Wang, Michiki Kurosawa+2 more|arXiv (Cornell University)|Nov 4, 2020

Topic Modeling19 references4 citations

TL;DR

This paper proposes two BERT-based methods—BERT-encoder and BERT-fused—for Chinese grammatical error correction (GEC), integrating a Chinese-RoBERTa-wwm-ext pre-trained model into a Transformer encoder-decoder framework. The BERT-encoder method achieves an F₀.₅ score of 35.51 with a 4-ensemble model, significantly outperforming the top team in the NLPCC 2018 shared task, while error analysis reveals sentence-level errors remain a major challenge.

ABSTRACT

In recent years, pre-trained models have been extensively studied, and several downstream tasks have benefited from their utilization. In this study, we verify the effectiveness of two methods that incorporate a BERT-based pre-trained model developed by Cui et al. (2020) into an encoder-decoder model on Chinese grammatical error correction tasks. We also analyze the error type and conclude that sentence-level errors are yet to be addressed.

Motivation & Objective

• To improve Chinese grammatical error correction (GEC) performance using BERT-based pre-trained models.
• To investigate the effectiveness of two integration strategies—encoder initialization and feature fusion—of a pre-trained model into a sequence-to-sequence GEC model.
• To analyze error types in Chinese GEC and identify persistent challenges, particularly sentence-level errors.
• To compare the performance of BERT-enhanced models against baseline and state-of-the-art models on the NLPCC 2018 GEC benchmark.

Proposed method

• Fine-tune a Transformer-based encoder-decoder model initialized with the Chinese-RoBERTa-wwm-ext pre-trained model weights in the encoder (BERT-encoder method).
• Use the pre-trained model to extract contextual representations as additional features for the decoder (BERT-fused method), following Zhu et al. (2020).
• Apply whole word masking (WWM) during pre-training to improve representation learning for Chinese characters and words.
• Train the models on the NLPCC 2018 GEC dataset, which includes 1 million training sentences and 2,000 test sentences from the PKU Chinese Learner Corpus.
• Use F₀.₅ score as the primary evaluation metric, with additional analysis on error type detection and correction performance.
• Conduct error type annotation on 100 development sentences to categorize errors into B (character-level), CC/CQ/CD (word-level), and CJ (sentence-level).

Figure 1: Two methods for incorporating a pre-trained model into the GEC model.

Experimental results

Research questions

• RQ1 Can BERT-based pre-trained models improve Chinese grammatical error correction performance when integrated into an encoder-decoder framework?
• RQ2 How do the BERT-encoder and BERT-fused methods compare in terms of error correction and detection across different error types?
• RQ3 What is the distribution of error types in Chinese GEC, and which types remain most challenging to correct?
• RQ4 Does the performance of the BERT-fused model depend on the quality of the initial model used for feature extraction?

Key findings

• The BERT-encoder method achieved an F₀.₅ score of 35.51 using a 4-ensemble model, surpassing the top-performing team in the NLPCC 2018 shared task by a large margin.
• The single BERT-encoder and BERT-fused models achieved F₀.₅ scores of 29.76 and 29.94, respectively, outperforming the baseline and approaching the performance of top teams.
• Sentence-level errors (CJ) were the most challenging, with both models showing significantly lower performance on this error type compared to word-level and character-level errors.
• The BERT-encoder model performed best on character-level errors (B), while the BERT-fused model showed stronger performance on word-level errors (CC, CQ, CD).
• The BERT-fused model’s performance was highly sensitive to the initial model; performance dropped when fine-tuned from a weak baseline, suggesting it benefits from a strong warm-up model.
• The models achieved higher recall than precision compared to top teams, indicating they are more effective at detecting errors but less precise in selecting the correct edits.