[Paper Review] Automatic Evaluation and Analysis of Idioms in Neural Machine Translation

Christos Baziotis, Prashant Mathur+1 more|arXiv (Cornell University)|Oct 10, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces LitTER, a novel automatic metric for detecting literal translation errors in neural machine translation (NMT) without requiring hand-crafted blocklists. Using monolingual pretraining, the authors show significant improvements in idiom translation even in zero-shot settings, demonstrating that pretraining enhances contextual understanding and reduces literal translation errors.

ABSTRACT

A major open problem in neural machine translation (NMT) is the translation of idiomatic expressions, such as "under the weather". The meaning of these expressions is not composed by the meaning of their constituent words, and NMT models tend to translate them literally (i.e., word-by-word), which leads to confusing and nonsensical translations. Research on idioms in NMT is limited and obstructed by the absence of automatic methods for quantifying these errors. In this work, first, we propose a novel metric for automatically measuring the frequency of literal translation errors without human involvement. Equipped with this metric, we present controlled translation experiments with models trained in different conditions (with/without the test-set idioms) and across a wide range of (global and targeted) metrics and test sets. We explore the role of monolingual pretraining and find that it yields substantial targeted improvements, even without observing any translation examples of the test-set idioms. In our analysis, we probe the role of idiom context. We find that the randomly initialized models are more local or "myopic" as they are relatively unaffected by variations of the idiom context, unlike the pretrained ones.

Motivation & Objective

• Address the lack of automatic, targeted evaluation methods for literal translation errors in NMT, especially for idioms.
• Overcome the reliance on human-annotated blocklists in prior methods for detecting literal translation errors.
• Investigate the impact of monolingual pretraining on improving idiom translation performance in zero-shot and few-shot settings.
• Analyze how contextual variation affects model behavior in translating idioms, comparing pre-trained and randomly initialized models.
• Develop a unified evaluation framework that combines LitTER with alignment-based metrics to comprehensively assess idiom translation quality.

Proposed method

• Propose LitTER, a metric that automatically generates word-level blocklists from bilingual dictionaries and filters them using reference translations to identify literal translation errors.
• Use MUSE bilingual word embeddings to translate individual words of a source idiom into the target language, forming an initial candidate blocklist.
• Filter the blocklist by removing words that appear in the reference translation to avoid false positives when the correct translation is literal.
• Apply LitTER to evaluate hypotheses by checking if any remaining blocklisted words appear in the translation output.
• Conduct controlled NMT experiments with models trained on different data splits (with/without test-set idioms) and using mBART for monolingual pretraining.
• Perform probing experiments by varying the context around idioms to assess model sensitivity and uncertainty in translation decisions.

Experimental results

Research questions

• RQ1 Can we automatically detect literal translation errors in NMT without relying on hand-crafted blocklists?
• RQ2 To what extent does monolingual pretraining improve the translation of idioms, even when no parallel examples of the test-set idioms are seen during training?
• RQ3 How does the context surrounding an idiom influence the model’s translation behavior, and does pretraining make models more sensitive to contextual variation?
• RQ4 How do pre-trained models compare to randomly initialized models in terms of contextual awareness and uncertainty when translating idioms?
• RQ5 Can LitTER be effectively combined with alignment-based metrics to provide a more comprehensive evaluation of idiom translation quality?

Key findings

• LitTER successfully detects literal translation errors without requiring language-specific hand-crafted blocklists, enabling scalable and automatic evaluation.
• Monolingual pretraining with mBART leads to substantial targeted improvements in idiom translation, even in zero-shot settings where no parallel examples of the test idioms are seen during training.
• Pre-trained models show greater sensitivity to contextual variation around idioms, indicating improved contextual understanding compared to randomly initialized models.
• Randomly initialized models are more 'myopic'—they are less affected by changes in idiom context, suggesting weaker contextual awareness.
• The frequency of literal translation errors is significantly reduced in pre-trained models, demonstrating that pretraining enhances non-compositional phrase understanding.
• LitTER is effective in isolating idiom-specific errors, and when combined with global metrics like BLEU, it provides a more nuanced evaluation of NMT performance on idioms.