[Paper Review] Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns

Lei Zhou, Liang Ding+1 more|arXiv (Cornell University)|Oct 10, 2020

Topic Modeling31 references11 citations

TL;DR

This paper proposes a zero-shot translation quality estimation (QE) method that improves BERTScore by incorporating explicit cross-lingual patterns—word alignment masking and generation score (perplexity)—to reduce token-level mismatching errors. The approach achieves performance comparable to supervised QE models, outperforming them on two of six translation directions in WMT 2020 QE shared task evaluations.

ABSTRACT

This paper describes our submission of the WMT 2020 Shared Task on Sentence Level Direct Assessment, Quality Estimation (QE). In this study, we empirically reveal the \ extit{mismatching issue} when directly adopting BERTScore to QE. Specifically, there exist lots of mismatching errors between the source sentence and translated candidate sentence with token pairwise similarity. In response to this issue, we propose to expose explicit cross-lingual patterns, \ extit{e.g.} word alignments and generation score, to our proposed zero-shot models. Experiments show that our proposed QE model with explicit cross-lingual patterns could alleviate the mismatching issue, thereby improving the performance. Encouragingly, our zero-shot QE method could achieve comparable performance with supervised QE method, and even outperforms the supervised counterpart on 2 out of 6 directions. We expect our work could shed light on the zero-shot QE model improvement.

Motivation & Objective

• To address the mismatching issue in BERTScore-based zero-shot QE, where source and target tokens are incorrectly aligned due to lack of cross-lingual guidance.
• To enhance zero-shot QE performance without requiring human-annotated references or fine-tuning.
• To investigate whether explicit cross-lingual signals—such as word alignments and generation scores—can improve the robustness and accuracy of reference-free QE.
• To demonstrate that a simple, non-fine-tuned BERTScore backbone can achieve competitive results when augmented with cross-lingual patterns.

Proposed method

• Adopts multilingual BERT and XLM to embed source and translated sentences into a shared semantic space for cross-lingual comparison.
• Applies greedy matching to align source tokens with the most similar target tokens based on dot-product similarity of contextual embeddings.
• Introduces cross-lingual alignment masking using GIZA++-derived word alignments to guide token matching and reduce mismatching errors.
• Incorporates a generation score (perplexity) from force-decoded target tokens using a pre-trained cross-lingual model to weight similarity scores.
• Combines alignment masking and perplexity scores with BERTScore via weighted fusion to improve matching accuracy.
• Uses Pearson and Kendall correlation with Direct Assessment (DA) scores as evaluation metrics on the WMT 2020 QE shared task.

Experimental results

Research questions

• RQ1 Can explicit cross-lingual patterns mitigate token-level mismatching in zero-shot QE when using BERTScore?
• RQ2 Does incorporating word alignment and generation score improve the correlation between predicted QE scores and human Direct Assessment (DA) scores?
• RQ3 Can a zero-shot QE model without fine-tuning outperform supervised baselines in certain translation directions?
• RQ4 How effective is the proposed method across diverse language pairs, especially low-resource ones?

Key findings

• The proposed method with both alignment masking and perplexity scoring achieved a Pearson correlation of 0.099 on en-de, 0.189 on en-zh, and 0.332 on ru-en, outperforming the baseline BERTScore on these directions.
• On the ru-en direction, the model achieved a Pearson correlation of 0.332, surpassing the supervised baseline (0.146) on the same test set.
• The model outperformed the supervised baseline on two of six translation directions (ru-en and si-en), demonstrating strong zero-shot generalization.
• The integration of perplexity scores alone improved performance on most directions, with the highest Pearson correlation of 0.105 on en-de and 0.439 on ro-en.
• The ablation study confirmed that both alignment masking and perplexity scoring contribute independently to performance gains, with the combined method yielding the best results.
• The method significantly reduced mismatching errors—e.g., correctly aligning the Russian token 'Назва' to 'named' instead of 'The'—as demonstrated in qualitative examples.