[Paper Review] Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation

Eva Vanmassenhove, Dimitar Shterionov+1 more|arXiv (Cornell University)|Jan 30, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper investigates how algorithmic bias in machine translation (MT) systems reduces linguistic complexity, coining the phenomenon 'machine translationese'—a simplified, less diverse output. Using lexical and morphological diversity metrics across PB-SMT, LSTM, and Transformer models, it finds that all MT systems significantly reduce lexical and morphological richness compared to training data, with Transformers performing best but still underperforming on morphological diversity in richer languages like French and Spanish.

ABSTRACT

Recent studies in the field of Machine Translation (MT) and Natural Language Processing (NLP) have shown that existing models amplify biases observed in the training data. The amplification of biases in language technology has mainly been examined with respect to specific phenomena, such as gender bias. In this work, we go beyond the study of gender in MT and investigate how bias amplification might affect language in a broader sense. We hypothesize that the 'algorithmic bias', i.e. an exacerbation of frequently observed patterns in combination with a loss of less frequent ones, not only exacerbates societal biases present in current datasets but could also lead to an artificially impoverished language: 'machine translationese'. We assess the linguistic richness (on a lexical and morphological level) of translations created by different data-driven MT paradigms - phrase-based statistical (PB-SMT) and neural MT (NMT). Our experiments show that there is a loss of lexical and morphological richness in the translations produced by all investigated MT paradigms for two language pairs (EN<=>FR and EN<=>ES).

Motivation & Objective

• To investigate whether algorithmic bias in data-driven MT systems leads to a measurable loss of linguistic richness, particularly in lexical and morphological diversity.
• To examine the sociolinguistic implications of this reduction, framing it as 'machine translationese'—an artificially impoverished form of language.
• To compare the impact of different MT paradigms (PB-SMT, LSTM, Transformer) on linguistic diversity across multiple language pairs (EN↔FR, EN↔ES).
• To develop and apply novel, adapted, and automatic evaluation metrics for lexical sophistication, synonym frequency, and morphological diversity.
• To assess whether the observed diversity loss correlates with translation quality metrics like BLEU and TER.

Proposed method

• Adapted the Lexical Frequency Profile (LFP) from second language acquisition to assess lexical sophistication and diversity in MT outputs.
• Applied Shannon entropy and Simpson’s diversity index to measure morphological diversity on lemmas with two or more wordforms, excluding single-wordform lemmas to avoid bias.
• Proposed a novel automatic evaluation method to assess synonym frequency, capturing lexical richness beyond standard diversity metrics.
• Evaluated three MT architectures—phrase-based SMT (PB-SMT), LSTM-based seq2seq, and Transformer (TRANS)—across two language pairs: English–French and English–Spanish.
• Computed diversity scores (H for Shannon entropy, D for Simpson’s diversity) on both original training data and MT-generated translations, normalizing to [0–100] for comparability.
• Used standard MT evaluation metrics (BLEU, TER) to correlate linguistic diversity with translation quality, enabling cross-validation of findings.

Experimental results

Research questions

• RQ1 To what extent do data-driven MT systems reduce lexical and morphological diversity compared to their training data?
• RQ2 How does the choice of MT architecture (PB-SMT, LSTM, Transformer) influence the preservation of linguistic richness in translations?
• RQ3 Does the loss of morphological diversity impair the system’s ability to generate grammatically correct forms, especially in morphologically rich languages?
• RQ4 Is there a measurable correlation between linguistic diversity and standard MT quality metrics like BLEU and TER?
• RQ5 To what extent does 'machine translationese' reflect algorithmic bias, and what are its long-term sociolinguistic implications?

Key findings

• All MT systems—PB-SMT, LSTM, and Transformer—significantly reduce lexical and morphological diversity compared to the original training data across both language pairs.
• Shannon entropy (H) and Simpson’s diversity (D) scores for morphological diversity were consistently lower in MT outputs: for French, H dropped from 75.20 (original) to 73.13 (Transformer), and D from 56.42 to 57.70; for Spanish, H fell from 78.42 to 77.23, and D from 54.96 to 56.26.
• Transformer models outperformed PB-SMT and LSTM in preserving morphological diversity, especially in translations into French and Spanish, though still below original data levels.
• For translations into English, PB-SMT outperformed LSTM in morphological diversity, suggesting architecture-specific effects depending on target language typology.
• The reduction in diversity was most pronounced in morphologically rich languages (French and Spanish), where the gap between original and MT output was larger than in English.
• There was a strong correlation between higher linguistic diversity and better BLEU/TER scores, suggesting that diversity metrics may serve as meaningful proxies for translation quality.