[Paper Review] "Bilingual Expert" Can Find Translation Errors

Kai Fan, Jiayi Wang+4 more|arXiv (Cornell University)|Jul 25, 2018

Natural Language Processing Techniques20 references3 citations

TL;DR

This paper proposes a novel neural bilingual expert model—a conditional target language model built with a bidirectional transformer—pre-trained on large parallel corpora to extract joint source-target representations for translation quality estimation (QE) without golden references. By combining these features with 4D mis-matching features and a Bi-LSTM predictor, the method achieves state-of-the-art performance on WMT 2017/2018 QE benchmarks, demonstrating strong zero-reference QE and potential for unified quality estimation and automatic post-editing.

ABSTRACT

Recent advances in statistical machine translation via the adoption of neural sequence-to-sequence models empower the end-to-end system to achieve state-of-the-art in many WMT benchmarks. The performance of such machine translation (MT) system is usually evaluated by automatic metric BLEU when the golden references are provided for validation. However, for model inference or production deployment, the golden references are prohibitively available or require expensive human annotation with bilingual expertise. In order to address the issue of quality evaluation (QE) without reference, we propose a general framework for automatic evaluation of translation output for most WMT quality evaluation tasks. We first build a conditional target language model with a novel bidirectional transformer, named neural bilingual expert model, which is pre-trained on large parallel corpora for feature extraction. For QE inference, the bilingual expert model can simultaneously produce the joint latent representation between the source and the translation, and real-valued measurements of possible erroneous tokens based on the prior knowledge learned from parallel data. Subsequently, the features will further be fed into a simple Bi-LSTM predictive model for quality evaluation. The experimental results show that our approach achieves the state-of-the-art performance in the quality estimation track of WMT 2017/2018.

Motivation & Objective

• To address the challenge of translation quality estimation (QE) without access to golden reference translations, which are costly to obtain in real-world deployment.
• To develop a general-purpose framework for automatic QE that leverages large-scale parallel corpora as prior knowledge.
• To unify quality estimation and automatic post-editing (APE) by training a conditional language model on gap prediction (insertions/deletions) for low-quality translations.
• To improve QE performance through a combination of pre-trained bilingual features and learned mis-matching features between predicted and actual translation tokens.
• To demonstrate the feasibility of using a single pre-trained model for both QE and APE, reducing reliance on separate, task-specific models.

Proposed method

• Pre-train a neural bilingual expert model using a bidirectional transformer decoder on large parallel monolingual corpora to learn joint source-target latent representations.
• Use the pre-trained model to generate conditional probability distributions for target tokens given source sentences, enabling gap prediction for missing or erroneous tokens.
• Extract 4-dimensional mis-matching features by comparing the model’s predicted token distribution with the actual machine translation output at each token position.
• Feed the extracted features—both model-derived and mis-matching—into a simple Bi-LSTM architecture for end-to-end quality estimation.
• Support BPE tokenization by using a learnable sparse matrix to average subword features per word, enabling differentiable backpropagation through segmentation.
• Extend the framework to support joint training of QE and APE by modeling insertion/deletion operations via gap prediction, with future work on end-to-end optimization.

Experimental results

Research questions

• RQ1 Can a pre-trained conditional language model based on parallel data serve as a robust prior for zero-reference translation quality estimation?
• RQ2 How effective are 4D mis-matching features in detecting translation errors without reference translations?
• RQ3 Can the same pre-trained bilingual model be used to support both quality estimation and automatic post-editing?
• RQ4 Does BPE tokenization improve QE performance, and how can it be integrated into the feature extraction pipeline with differentiable operations?
• RQ5 Can a unified framework jointly optimize quality estimation and automatic post-editing using a single pre-trained model?

Key findings

• The proposed method achieves state-of-the-art performance on the WMT 2017 and 2018 translation quality estimation benchmarks across most public datasets.
• Even with only the 4D mis-matching features and no additional model fine-tuning, the approach achieves competitive results, demonstrating the robustness of the feature design.
• The use of BPE tokenization improves QE performance on word-level labeling tasks, especially when threshold tuning is applied to the prediction scores.
• The pre-trained bilingual expert model enables gap prediction for missing tokens, which can be directly used for automatic post-editing, suggesting a unified path for QE and APE.
• The framework supports differentiable feature extraction under BPE tokenization, enabling potential joint training of the QE and APE components in the future.
• The model's ability to detect errors in translations without golden references makes it suitable for real-world deployment in production MT systems.