[Paper Review] Regressive Ensemble for Machine Translation Quality Evaluation

Michal Štefánik, Vít Novotný+1 more|arXiv (Cornell University)|Sep 15, 2021

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper proposes RegEMT, a regressive ensemble model that combines diverse machine translation evaluation metrics—surface, syntactic, and semantic—into a single linear regression framework to improve correlation with expert MQM scores. It achieves state-of-the-art performance across monolingual and zero-shot cross-lingual settings, with a novel reference-free baseline (Reg-base) outperforming BLEU and METEOR, and demonstrates strong transferability to unseen languages.

ABSTRACT

This work introduces a simple regressive ensemble for evaluating machine translation quality based on a set of novel and established metrics. We evaluate the ensemble using a correlation to expert-based MQM scores of the WMT 2021 Metrics workshop. In both monolingual and zero-shot cross-lingual settings, we show a significant performance improvement over single metrics. In the cross-lingual settings, we also demonstrate that an ensemble approach is well-applicable to unseen languages. Furthermore, we identify a strong reference-free baseline that consistently outperforms the commonly-used BLEU and METEOR measures and significantly improves our ensemble's performance.

Motivation & Objective

• To improve machine translation quality evaluation by combining multiple metrics that capture surface, syntactic, and semantic properties of translations.
• To investigate whether an ensemble approach can generalize to unseen language pairs in zero-shot cross-lingual settings.
• To evaluate the performance of reference-free surface-level metrics as a baseline, especially in comparison to established reference-based metrics.
• To assess the impact of contextualized versus decontextualized token representations on evaluation performance.
• To reduce blind spots in individual metrics by leveraging the orthogonality of diverse evaluation signals through ensemble learning.

Proposed method

• Constructs a regressive ensemble (RegEMT) by training a linear regression model on a combination of 15+ metrics, including BLEU, METEOR, BERTScore, WMD, Prism, and novel metrics.
• Employs a stepwise backward elimination process to iteratively remove metrics with the lowest contribution to correlation with MQM scores.
• Introduces Reg-base, a reference-free baseline using only source and reference length features derived from a multilingual WordPiece tokenizer.
• Uses FastText and BERT-based contextual embeddings for semantic matching, with WMD and soft cosine measures for sentence-level similarity.
• Applies the ensemble model on the WMT 2021 MQM dataset, using averaged human judgments as the gold standard.
• Performs model training and evaluation using a train/test split based on unique source texts, with open-source code and Docker deployment for reproducibility.

Experimental results

Research questions

• RQ1 Can a regressive ensemble of diverse metrics significantly improve correlation with expert MQM scores compared to single metrics?
• RQ2 Can such an ensemble generalize effectively to zero-shot cross-lingual settings, including unseen language pairs?
• RQ3 Can a simple, reference-free surface-level metric (Reg-base) outperform established metrics like BLEU and METEOR?
• RQ4 Does the use of contextualized embeddings provide a consistent advantage over decontextualized or TF-IDF-enhanced embeddings in evaluation performance?
• RQ5 To what extent do the individual metrics in the ensemble contribute to the final correlation, and which are most critical for performance?

Key findings

• The RegEMT ensemble achieves significantly higher Spearman correlation with MQM scores than any single metric across all language pairs, including en-de, en-cs, and cross-lingual zero-shot setups.
• The reference-free Reg-base baseline consistently outperforms BLEU and METEOR, demonstrating that length-based features from multilingual tokenizers can serve as a strong foundation for evaluation.
• In ablation studies, the ensemble maintains high performance even when removing less correlated metrics, with WMD-decontextualized-tfidf and Prism being the last to be eliminated in en-de.
• The ensemble shows strong transferability, maintaining high performance on unseen language pairs, indicating that multilingual metrics contribute meaningfully to generalization.
• Contextualized metrics such as BERTScore and WMD-contextual contribute most significantly to the ensemble’s performance, while decontextualized versions show mixed results, with some (e.g., WMD-decontextualized-tfidf) improving by 16–18% over their contextualized counterparts.
• The ensemble inherits systematic biases from its components, as seen in the en-de pair where low correlations of individual metrics propagate to the ensemble, highlighting a key limitation despite overall gains.