[Paper Review] Word-Alignment-Based Segment-Level Machine Translation Evaluation using Word Embeddings

Junki Matsuo, Mamoru Komachi+1 more|arXiv (Cornell University)|Apr 2, 2017

Natural Language Processing Techniques7 references3 citations

TL;DR

This paper proposes word-alignment-based segment-level machine translation evaluation using word embeddings, leveraging three alignment strategies—average, maximum, and Hungarian—to compute sentence similarity. It outperforms prior word embedding-based methods, with maximum alignment similarity showing the highest human correlation on European–English datasets and average alignment on Japanese–English datasets.

ABSTRACT

One of the most important problems in machine translation (MT) evaluation is to evaluate the similarity between translation hypotheses with different surface forms from the reference, especially at the segment level. We propose to use word embeddings to perform word alignment for segment-level MT evaluation. We performed experiments with three types of alignment methods using word embeddings. We evaluated our proposed methods with various translation datasets. Experimental results show that our proposed methods outperform previous word embeddings-based methods.

Motivation & Objective

• Address the challenge of segment-level MT evaluation where surface-form differences obscure semantic equivalence.
• Improve upon BLEU and METEOR by incorporating semantic similarity via word embeddings instead of relying on n-gram matching or external resources.
• Develop a lightweight, corpus-based evaluation metric that avoids costly annotations and complex models.
• Investigate how different word alignment heuristics affect MT evaluation performance across diverse language pairs.
• Optimize alignment thresholds and alignment strategies to maximize correlation with human judgments.

Proposed method

• Use pre-trained word2vec embeddings to compute cosine similarity between words in hypothesis and reference sentences.
• Apply three word-alignment heuristics: 1:1 (Hungarian), 1:n (maximum), and m:n (average) to match words across sentences.
• Introduce a similarity threshold (e.g., 0.2) to filter out noisy or low-similarity word alignments.
• Compute sentence-level similarity as the average or maximum of aligned word similarities, depending on the method.
• Apply length penalty adjustments via alignment type to compensate for length discrepancies between hypothesis and reference.
• Train and evaluate using only a raw monolingual corpus (Google News) for word embeddings, avoiding external linguistic resources.

Experimental results

Research questions

• RQ1 Can word-alignment-based sentence similarity using word embeddings improve segment-level MT evaluation beyond surface-level n-gram matching?
• RQ2 How do different alignment strategies (average, maximum, 1:1) affect correlation with human judgments across diverse language pairs?
• RQ3 Does thresholding low-similarity alignments improve robustness and correlation in MT evaluation?
• RQ4 How does the performance of word-alignment-based metrics vary between morphologically simple (e.g., English) and complex (e.g., Japanese) languages?
• RQ5 Can a lightweight, embedding-based metric achieve higher human correlation than established metrics like BLEU and DREEM without external annotations?

Key findings

• Maximum alignment similarity (MAS) achieved the highest Kendall’s τ correlation (0.373) on WMT15 European–English datasets, outperforming DREEM and BLEU.
• Average alignment similarity (AAS) achieved the highest correlation (0.343) on NTCIR8 Japanese–English datasets, indicating better robustness for morphologically complex language pairs.
• For WMT datasets, a threshold of 0.2 maximized MAS performance, demonstrating sensitivity to alignment filtering.
• Hungarian (1:1) alignment showed consistently low performance (e.g., τ = 0.106 on WMT12), suggesting over-constraint reduces effectiveness.
• AAS with high thresholds (0.6–0.9) showed stable, high correlation across Japanese–English datasets, indicating reliability for low-resource or complex language pairs.
• The method achieved better human correlation than DREEM on all European–English language pairs except Czech–English, confirming its effectiveness in semantic similarity modeling.