[论文解读] LEPOR: An Augmented Machine Translation Evaluation Metric

Lifeng Han|arXiv (Cornell University)|Mar 26, 2017

Natural Language Processing Techniques参考文献 8被引用 8

一句话总结

LEPOR 是一种新颖的增强型机器翻译评估指标，通过将可调校、基于语言学的特征（尤其是词性标注）整合到灵活且经过优化的评分框架中，提升了在多样化语言对上的准确性和鲁棒性。该指标在跨语言评估中表现优于现有指标，尤其在非英语源语言场景下表现突出，并在 ACL-WMT 共享任务中展现出优异性能，同时在无需参考译文的质量估计方面，应用了 NB、SVM 和 CRF 模型，取得额外贡献。

ABSTRACT

Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whether the translation system makes an improvement or not. The traditional manual judgment methods are expensive, time-consuming, unrepeatable, and sometimes with low agreement. On the other hand, the popular automatic MT evaluation methods have some weaknesses. Firstly, they tend to perform well on the language pairs with English as the target language, but weak when English is used as source. Secondly, some methods rely on many additional linguistic features to achieve good performance, which makes the metric unable to replicate and apply to other language pairs easily. Thirdly, some popular metrics utilize incomprehensive factors, which result in low performance on some practical tasks. In this thesis, to address the existing problems, we design novel MT evaluation methods and investigate their performances on different languages. Firstly, we design augmented factors to yield highly accurate evaluation. Secondly, we design a tunable evaluation model where weighting of factors can be optimized according to the characteristics of languages. Thirdly, in the enhanced version of our methods, we design concise linguistic feature using part-of-speech (POS) to show that our methods can yield even higher performance when using some external linguistic resources. Finally, we introduce the practical performance of our metrics in the ACL-WMT workshop shared tasks, which show that the proposed methods are robust across different languages. In addition, we also present some novel work on quality estimation of MT without using reference translations including the usage of probability models of Naïve Bayes (NB), support vector machine (SVM) classification algorithms, and CRFs.

研究动机与目标

• 为解决现有自动 MT 评估指标的局限性，特别是其对英语目标译文的偏见以及对复杂、不可复现的语言学特征的依赖。
• 开发一种可调校的评估模型，根据语言特异性特征动态调整评估因子的权重。
• 通过简洁的语言学特征（如词性标注）提升性能，增强在不同语言对之间的泛化能力与可复现性。
• 实现稳健的基于参考译文与无需参考译文的 MT 评估，包括利用朴素贝叶斯、SVM 和 CRF 等概率模型进行质量估计。

提出的方法

• 引入基于语言学分析的增强型评估因子，包括词性标注，以提升对齐与流畅性评估能力。
• 采用可调校模型，针对每对语言优化特征权重，以反映语言差异并提升与人类判断的相关性。
• 将词性（POS）特征作为轻量级、高效的语言学资源，提升性能而不增加复杂度。
• 应用概率模型（朴素贝叶斯、SVM 和 CRF）实现无需参考译文的质量估计，直接从系统输出预测 MT 质量。
• 在 ACL-WMT 共享任务中验证该指标，证明其在多样化语言对中的一致性表现。
• 采用混合方法，结合自动评估指标设计与质量估计，支持基于参考译文与无需参考译文的双重评估。

实验结果

研究问题

• RQ1 可调校、增强型指标是否能在多种语言对中提升 MT 评估性能，特别是在英语为源语言的情况下？
• RQ2 轻量级语言学特征（如词性标注）在提升 MT 评估准确性和泛化能力方面效果如何？
• RQ3 基于 NB、SVM 和 CRF 的无需参考译文质量估计模型在无参考译文条件下能达到多大程度的可靠性能？
• RQ4 所提出的指标是否能在包括低资源语言和非英语源语言在内的多个语言对中保持与人类判断的高相关性？
• RQ5 与固定权重指标相比，优化特征权重的集成方式在提升鲁棒性与可复现性方面有何优势？

主要发现

• LEPOR 在 ACL-WMT 共享任务的多个语言对中表现优异，展现出强鲁棒性与人类判断的高相关性。
• 引入词性（POS）特征显著提升了评估准确性，尤其在低资源或非英语源语言场景下效果明显。
• 可调校权重机制使模型能有效适应语言特异性特征，从而增强整体评估的可靠性。
• 基于 NB、SVM 和 CRF 的无需参考译文质量估计模型表现良好，为无参考译文场景提供了可行替代方案。
• 在英语作为源语言的场景下，该指标优于传统自动评估指标，有效解决了现有系统的关键局限。
• 该方法具备可复现性与可扩展性，设计原则清晰，支持在不需大量重新训练的情况下应用于新语言对。