[Paper Review] Investigating Backtranslation in Neural Machine Translation

Alberto Poncelas, Dimitar Shterionov+3 more|arXiv (Cornell University)|Apr 17, 2018

Natural Language Processing TechniquesComputer Science23 references88 citations

TL;DR

The paper systematically evaluates how back-translated (synthetic) data, alone or with authentic data, affects German→English NMT performance. It finds that synthetic data can approach authentic data in quality, hybrid data often yields the best results, and a tipping point exists where too much synthetic data degrades performance.

ABSTRACT

A prerequisite for training corpus-based machine translation (MT) systems -- either Statistical MT (SMT) or Neural MT (NMT) -- is the availability of high-quality parallel data. This is arguably more important today than ever before, as NMT has been shown in many studies to outperform SMT, but mostly when large parallel corpora are available; in cases where data is limited, SMT can still outperform NMT. Recently researchers have shown that back-translating monolingual data can be used to create synthetic parallel corpora, which in turn can be used in combination with authentic parallel data to train a high-quality NMT system. Given that large collections of new parallel text become available only quite rarely, backtranslation has become the norm when building state-of-the-art NMT systems, especially in resource-poor scenarios. However, we assert that there are many unknown factors regarding the actual effects of back-translated data on the translation capabilities of an NMT model. Accordingly, in this work we investigate how using back-translated data as a training corpus -- both as a separate standalone dataset as well as combined with human-generated parallel data -- affects the performance of an NMT model. We use incrementally larger amounts of back-translated data to train a range of NMT systems for German-to-English, and analyse the resulting translation performance.

Motivation & Objective

• Assess the impact of back-translated data on NMT performance across data sizes

Proposed method

• Train 2-layer LSTM NMT models with attention using OpenNMT-py on WMT 2015 German→English data
• Create synthetic data by back-translating English→German with an initial NMT model
• Evaluate authentic-only, synthetic-only, and hybrid datasets using BLEU, TER, METEOR, and CHRF1
• Incrementally vary authentic and synthetic data sizes to observe performance trends
• Use a fixed baseline configuration to isolate effects of backtranslated data

Experimental results

Research questions

• RQ1 How does increasing the amount of back-translated data affect NMT quality when trained with authentic data?
• RQ2 Can synthetic data alone produce competitive NMT performance compared to authentic data?
• RQ3 What is the optimal balance between authentic and back-translated data for hybrid training?
• RQ4 Is there a tipping point where adding more synthetic data harms hybrid model performance?

Key findings

• NMT models improve with more authentic data, peaking at 3M authentic sentences (BLEU 0.2446) before a slight drop at 3.5M
• Synthetic-only models can approach authentic-data performance, peaking at BLEU 0.2363 with 3.5M synthetic sentences
• Hybrid models (1M authentic + varying synthetic data) generally outperform authentic-only baselines at small to mid data sizes
• A synthetic-to-authentic ratio up to 2:1 in hybrids showed gains, but beyond that the benefit plateaus and can decline (e.g., 3.5M hybrid BLEU 0.2442 vs 3M hybrid BLEU 0.2442)
• Back-translation remains beneficial in resource-poor settings, enabling large synthetic corpora to bootstrap NMT systems