[Paper Review] Synthetic and Natural Noise Both Break Neural Machine Translation

Yonatan Belinkov, Yonatan Bisk|arXiv (Cornell University)|Nov 6, 2017

Natural Language Processing TechniquesComputer Science42 references399 citations

TL;DR

The paper shows that state-of-the-art character-based NMT models are brittle to both synthetic and natural noise, and proposes structure-invariant representations and adversarial training (noise-robust charCNN) to improve robustness.

ABSTRACT

Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems. Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.

Motivation & Objective

• Assess how synthetic and natural noise affect state-of-the-art character-based NMT systems.
• Investigate robustness strategies for NMT under noisy inputs.
• Evaluate structure-invariant representations and adversarial training for noise resilience in MT.
• Compare performance across multiple language pairs and noise types.

Proposed method

• Evaluate three NMT architectures with character information (char2char, Nematus with BPE, and a word representation via charCNN).
• Generate and inject natural and synthetic noise into source sentences (Swap, Middle Random, Fully Random, Keyboard Typo, plus natural errors from corpora).
• Test robustness by BLEU on clean vs. noisy inputs and analyze failures.
• Explore structure-invariant word representations (meanChar) and train models on noisy data (black-box adversarial training) and mixed-noise ensembles.
• Provide release of noise generation code and data for reproducibility.

Experimental results

Research questions

• RQ1 How do synthetic and natural word-level/character-level noises impact translation quality across languages?
• RQ2 Can structure-invariant representations or adversarial training improve robustness of character-based NMT models to various noise types?
• RQ3 What are the limitations of synthetic noise in reflecting natural human errors for MT robustness?
• RQ4 Do mixed-noise training regimes yield broader robustness than single-noise training?

Key findings

• All models suffer substantial BLEU drops when tested on noisy texts, including natural noise.
• CharCNN with noise-robust training can address multiple noise types better than others, though natural noise remains challenging.
• MeanChar representations are robust to scrambling but underperform on other noise types; they are not consistently robust across languages.
• Training on specific noise types improves test-time performance for that noise, but not universally across noise families; mixed-noise training (Rand+Key+Nat) yields best average robustness.
• Natural noise often stems from phonetic/phonological errors and omissions not fully captured by synthetic noise, explaining gaps in robustness.