[Paper Review] Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation

Eva Vanmassenhove, Dimitar Shterionov+1 more|arXiv (Cornell University)|Jun 28, 2019

Natural Language Processing Techniques25 references48 citations

TL;DR

The paper empirically quantifies lexical richness loss in MT versus HT across EN–FR and EN–ES, showing MTs favor frequent words, reduce diversity, and exhibit potential algorithmic bias.

ABSTRACT

This work presents an empirical approach to quantifying the loss of lexical richness in Machine Translation (MT) systems compared to Human Translation (HT). Our experiments show how current MT systems indeed fail to render the lexical diversity of human generated or translated text. The inability of MT systems to generate diverse outputs and its tendency to exacerbate already frequent patterns while ignoring less frequent ones, might be the underlying cause for, among others, the currently heavily debated issues related to gender biased output. Can we indeed, aside from biased data, talk about an algorithm that exacerbates seen biases?

Motivation & Objective

• Assess whether neural and statistical MT systems preserve lexical richness compared to human translations.
• Quantify how MT architectures (RNN, Transformer, SMT) influence word frequencies and diversity.
• Investigate the relation between training data bias, back-translation, and output diversity.
• Explore implications for gender bias and morphologically rich languages due to lexical loss.

Proposed method

• Train and evaluate three MT architectures (RNN, Transformer, SMT) on EN–FR and EN–ES using Europarl data.
• Use original and back-translated data to create FF, BACK, and REV systems for cross-analysis.
• Measure lexical diversity with four metrics (TTR, Yule’s I, MTLD, plus a derived 1000x scaled form).
• Analyze word-frequency bias by classifying words into six frequency-change categories relative to HT.
• Compare seen (training data) vs unseen data to separate algorithmic effects from data effects.

Experimental results

Research questions

• RQ1 Does neural MT (NMT) and SMT differ in preserving lexical diversity compared to human translations?
• RQ2 Do MT systems bias output toward more frequent words, at the expense of rarer ones?
• RQ3 How does back-translation/backing data influence lexical richness and bias propagation?
• RQ4 Are observed effects consistent across language pairs EN–FR and EN–ES and across seen vs unseen data?

Key findings

• MT systems generally under-represent lexical diversity compared with HT across both language pairs.
• SMT tends to retain more lexical richness by some metrics (TTR, Yule’s I) than neural methods, while Transformer shows higher richness on MTLD.
• All MT variants bias toward frequent HT words and disproportionately drop less frequent words, sometimes eliminating them entirely.
• Back-translation reduces vocabulary size and can amplify bias or alter learned word associations.
• Seen vs unseen data show similar trends in lexical richness loss, with slightly stronger effects on unseen data.