[Paper Review] Sometimes We Want Translationese

Prasanna Parthasarathi, Koustuv Sinha+2 more|arXiv (Cornell University)|Apr 15, 2021

Natural Language Processing Techniques64 references4 citations

TL;DR

This paper introduces a framework to evaluate neural machine translation (NMT) systems on their trade-off between faithfulness and robustness to word-order perturbations in the source input. Using novel perturbation functions and metrics, it finds that state-of-the-art Transformer-based models (Helsinki/OPUS) strongly favor robustness over faithfulness across seven languages, often producing hallucinated translations even when input word order is altered, highlighting a critical trade-off in NMT design.

ABSTRACT

Rapid progress in Neural Machine Translation (NMT) systems over the last few years has been driven primarily towards improving translation quality, and as a secondary focus, improved robustness to input perturbations (e.g. spelling and grammatical mistakes). While performance and robustness are important objectives, by over-focusing on these, we risk overlooking other important properties. In this paper, we draw attention to the fact that for some applications, faithfulness to the original (input) text is important to preserve, even if it means introducing unusual language patterns in the (output) translation. We propose a simple, novel way to quantify whether an NMT system exhibits robustness and faithfulness, focusing on the case of word-order perturbations. We explore a suite of functions to perturb the word order of source sentences without deleting or injecting tokens, and measure the effects on the target side in terms of both robustness and faithfulness. Across several experimental conditions, we observe a strong tendency towards robustness rather than faithfulness. These results allow us to better understand the trade-off between faithfulness and robustness in NMT, and opens up the possibility of developing systems where users have more autonomy and control in selecting which property is best suited for their use case.

Motivation & Objective

• To investigate whether NMT systems prioritize faithfulness to the original input or robustness to input perturbations, especially word-order changes.
• To identify cases where preserving input structure—such as in poetry, language learning, or syntactic analysis—is more valuable than producing fluent but altered translations.
• To develop and validate new metrics and perturbation functions for quantifying faithfulness and robustness in NMT outputs.
• To demonstrate that current SOTA NMT models exhibit a strong bias toward robustness, often at the cost of faithfulness, especially in non-English languages.

Proposed method

• Proposes three categories of word-order perturbation functions: dependency tree-based, POS-tag-based, and random shuffles, all preserving source tokens without deletion or insertion.
• Introduces two novel metrics: α_E (faithfulness) measuring word overlap between original and perturbed translations, and β_1, β_2 (robustness) measuring translation similarity between perturbed and original inputs.
• Employs the Helsinki/OPUS Transformer-based NMT model to translate from English into seven diverse languages: German, French, Spanish, Russian, Chinese, Japanese, and Italian.
• Applies perturbations across 16 unique functions and computes α_E and β scores across multiple experimental conditions to assess model behavior.
• Uses BLEU scores as a baseline for translation quality and correlates them with robustness and faithfulness metrics to assess model behavior.
• Analyzes the relationship between perturbation type, language family, and model performance, identifying patterns in robustness and faithfulness across languages and functions.

Experimental results

Research questions

• RQ1 To what extent do state-of-the-art NMT models preserve translation output when the source sentence’s word order is perturbed?
• RQ2 How does the model’s robustness compare to its faithfulness across different languages and perturbation types?
• RQ3 What is the relationship between translation quality (BLEU) and the model’s tendency toward robustness or faithfulness?
• RQ4 Which perturbation functions most strongly trigger robust or faithful behavior in NMT models?
• RQ5 Can we quantify and compare faithfulness and robustness in NMT systems to guide model selection for specific use cases?

Key findings

• All 7 target languages showed a consistent bias toward robustness, with faithfulness scores (α_E) consistently lower than robustness scores (β_1), indicating that models are more robust than faithful.
• The Japanese translation model exhibited the highest robustness (β_1) but also the lowest faithfulness (α_E), suggesting a strong tendency to normalize input structure.
• Models showed higher robustness on perturbations involving parts of speech (e.g., verbAtBeginning, nounVerbSwap), indicating that POS-based reordering increases robustness.
• The correlation between BLEU score and robustness (β_1) suggests that high-performing models are more likely to produce consistent outputs despite input perturbations.
• Certain perturbation functions, such as TreeMirrorIn and Reversed, led to higher faithfulness, while FunctionalShuffle induced the most robust behavior across all languages.
• A small fraction of translations showed β_1 > β, indicating rare cases where reordering the source improved translation quality, suggesting potential for probing model 'unnaturalness'.