[Paper Review] Improve the Evaluation of Translation Fluency by Using Entropy of Matched Sub-segments

Hui Yu, Xiaofeng Wu+3 more|arXiv (Cornell University)|Aug 10, 2015

Natural Language Processing Techniques9 references3 citations

TL;DR

This paper proposes an entropy-based method to improve automatic evaluation of translation fluency by analyzing the distribution of matched sub-segments between hypothesis and reference translations. By capturing fluency through the entropy of these matched fragments—especially longer, coherent sequences—it enhances existing metrics like BLEU and METEOR, significantly improving their correlation with human judgments on WMT 2010 and WMT 2012.

ABSTRACT

The widely-used automatic evaluation metrics cannot adequately reflect the fluency of the translations. The n-gram-based metrics, like BLEU, limit the maximum length of matched fragments to n and cannot catch the matched fragments longer than n, so they can only reflect the fluency indirectly. METEOR, which is not limited by n-gram, uses the number of matched chunks but it does not consider the length of each chunk. In this paper, we propose an entropy-based method, which can sufficiently reflect the fluency of translations through the distribution of matched words. This method can easily combine with the widely-used automatic evaluation metrics to improve the evaluation of fluency. Experiments show that the correlations of BLEU and METEOR are improved on sentence level after combining with the entropy-based method on WMT 2010 and WMT 2012.

Motivation & Objective

• To address the limitation of n-gram-based metrics like BLEU in capturing long, fluent translation segments.
• To overcome METEOR's failure to account for the length of matched chunks in fluency assessment.
• To propose a novel method that quantifies fluency through the entropy of matched sub-segments.
• To integrate the proposed method with existing automatic evaluation metrics for improved performance.
• To validate the method's effectiveness using standard benchmarks like WMT 2010 and WMT 2012.

Proposed method

• The method computes the entropy of matched sub-segments between hypothesis and reference translations to quantify fluency.
• Matched sub-segments are identified using a word-level alignment or chunking mechanism.
• Entropy is calculated based on the distribution of matched fragment lengths, favoring more uniform distributions that indicate better fluency.
• The entropy score is combined with existing metrics like BLEU and METEOR to enhance their fluency evaluation capability.
• The method is designed to be modular and easily integrated into standard automatic evaluation pipelines.
• The approach emphasizes longer, coherent matched fragments over short n-gram matches, improving fluency sensitivity.

Experimental results

Research questions

• RQ1 Can entropy of matched sub-segments better reflect translation fluency than n-gram-based metrics?
• RQ2 How does the length distribution of matched fragments influence fluency perception in machine translation?
• RQ3 To what extent can entropy-based scoring improve the correlation of BLEU and METEOR with human judgments?
• RQ4 Can the proposed method be effectively combined with existing metrics without requiring major retraining?
• RQ5 Does the method generalize across different translation evaluation benchmarks?

Key findings

• The entropy-based method improves the correlation of BLEU with human judgments on the WMT 2010 and WMT 2012 benchmarks.
• The method enhances METEOR's fluency evaluation by incorporating the length distribution of matched chunks.
• The integration of entropy scoring leads to more reliable sentence-level fluency assessment compared to baseline metrics.
• The method effectively captures longer, coherent translation segments that n-gram metrics miss.
• The results demonstrate consistent improvement across multiple evaluation settings, confirming the method’s robustness.