[Paper Review] Understanding Back-Translation at Scale

Sergey Edunov, Myle Ott+2 more|arXiv (Cornell University)|Aug 28, 2018

Natural Language Processing TechniquesComputer Science37 references111 citations

TL;DR

The paper analyzes back-translation for neural machine translation at scale, showing that sampling or noised beam-generated synthetic sources yield stronger training signals than beam/greedy MAP outputs, and achieving state-of-the-art results on WMT benchmarks using public data and large monolingual corpora.

ABSTRACT

An effective method to improve neural machine translation with monolingual data is to augment the parallel training corpus with back-translations of target language sentences. This work broadens the understanding of back-translation and investigates a number of methods to generate synthetic source sentences. We find that in all but resource poor settings back-translations obtained via sampling or noised beam outputs are most effective. Our analysis shows that sampling or noisy synthetic data gives a much stronger training signal than data generated by beam or greedy search. We also compare how synthetic data compares to genuine bitext and study various domain effects. Finally, we scale to hundreds of millions of monolingual sentences and achieve a new state of the art of 35 BLEU on the WMT'14 English-German test set.

Motivation & Objective

• Assess how different methods of generating synthetic source sentences affect back-translation in neural machine translation.
• Quantify the training signal and learning dynamics from various synthetic data generation strategies.
• Evaluate domain effects and compare synthetic data to genuine bitext in both high- and low-resource settings.
• Scale back-translation to hundreds of millions of monolingual sentences and benchmark against public data.

Proposed method

• Train a Transformer-based NMT model (Big Transformer) in a forward direction using real bitext and augmented with synthetic source sentences generated for back-translation.
• Compare synthetic-source generation methods: greedy, beam search (beam size 5), unrestricted sampling, restricted sampling (top-k with k=10), and beam with added noise.
• Inject synthetic data in varying amounts (e.g., up to tens of millions of sentence pairs) and adjust bitext upsampling during training.
• Measure training signals via cross-entropy perplexity on synthetic vs. bitext data and analyze training dynamics.
• Evaluate domain effects by contrasting BT data from monolingual news data versus real bitext.
• Scale experiments to large monolingual corpora (up to 226M sentences) and perform cross-dataset benchmarks (WMT’14 English-German, WMT’14 English-French, WMT’18 English-German).

Experimental results

Research questions

• RQ1 Does the method of generating synthetic sources (sampling vs beam/greedy) significantly affect NMT performance?
• RQ2 Does synthetic data provide a stronger training signal than real bitext, and under what resource conditions is this most pronounced?
• RQ3 How does the domain of monolingual data used for back-translation impact performance on in-domain vs. mixed-domain test sets?
• RQ4 Can back-translation with large-scale monolingual data reach or approach performance gains obtainable with real bitext?
• RQ5 How do scaling and upsampling of bitext vs. synthetic data influence final model accuracy?

Key findings

• Sampling and noisy beam generation consistently outperform argmax-based methods (greedy/beam) by about 0.8–1.1 BLEU on multiple English–German test sets.
• Synthetic data produced by sampling or adding noise provides a much stronger training signal than data from beam or greedy search, indicated by higher end-model BLEU and perplexity patterns.
• In some cases, synthetic data can match a large portion (up to 83%) of the gains provided by real bitext, especially when domains align (BT-news approaching BT-bitext in newstest2012).
• On WMT’14 English–German, back-translation with sampling plus 226M monolingual sentences achieves a new state-of-the-art of 35 BLEU (tokenized) using only public benchmark data; on WMT’14 English–French, 45.6 BLEU is reported (tokenized).
• Ensembling multiple BT models and filtering source copies contributed to strong WMT’18 English–German results, with substantial gains over baselines.
• Overall, BT with sampling/noised outputs yields a robust data augmentation strategy that scales well and competes with high-quality non-benchmark data.