[Paper Review] Cross-model Back-translated Distillation for Unsupervised Machine Translation

Xuan-Phi Nguyen, Shafiq Joty+3 more|arXiv (Cornell University)|Jun 3, 2020

Natural Language Processing Techniques33 references12 citations

TL;DR

This paper proposes Cross-model Back-translated Distillation (CBD), a novel method that enhances unsupervised machine translation by training two separate UMT models to iteratively back-translate monolingual data, generating diverse synthetic parallel data for distillation. CBD achieves state-of-the-art results on WMT’14 English-French (38.2 BLEU), WMT’16 English-German (30.1 BLEU), and English-Romanian (36.3 BLEU), with consistent gains of 1.5–3.3 BLEU on IWSLT benchmarks, outperforming existing methods through improved data diversity.

ABSTRACT

Recent unsupervised machine translation (UMT) systems usually employ three main principles: initialization, language modeling and iterative back-translation, though they may apply them differently. Crucially, iterative back-translation and denoising auto-encoding for language modeling provide data diversity to train the UMT systems. However, the gains from these diversification processes has seemed to plateau. We introduce a novel component to the standard UMT framework called Cross-model Back-translated Distillation (CBD), that is aimed to induce another level of data diversification that existing principles lack. CBD is applicable to all previous UMT approaches. In our experiments, CBD achieves the state of the art in the WMT'14 English-French, WMT'16 English-German and English-Romanian bilingual unsupervised translation tasks, with 38.2, 30.1, and 36.3 BLEU respectively. It also yields 1.5-3.3 BLEU improvements in IWSLT English-French and English-German tasks. Through extensive experimental analyses, we show that CBD is effective because it embraces data diversity while other similar variants do not.

Motivation & Objective

• To address the plateau in performance gains from existing unsupervised machine translation (UMT) methods despite extensive use of denoising auto-encoding and iterative back-translation.
• To introduce a new principle—Cross-model Back-translated Distillation (CBD)—that adds a higher level of data diversification beyond current UMT frameworks.
• To enhance model generalization and robustness by leveraging synthetic parallel data generated through cross-translation between two independently trained UMT agents.
• To demonstrate that CBD is effective, generalizable, and applicable to any existing UMT method without requiring large-scale pretraining or parallel data.
• To show that CBD improves performance not by model ensembling or direct distillation, but by generating more diverse and informative synthetic training data.

Proposed method

• CBD trains two separate bidirectional UMT agents using standard UMT principles, including initialization, denoising auto-encoding, and iterative back-translation.
• The first agent translates monolingual source language data into target language, producing synthetic target sentences.
• The second agent then back-translates these synthetic target sentences back into the source language, creating a second layer of synthetic parallel data.
• The resulting synthetic parallel pairs from both translation directions are used to distill a final supervised MT model, enhancing its generalization and robustness.
• CBD is applied as an external module to any existing UMT framework, making it compatible with diverse initialization and training strategies.
• The method avoids naive data augmentation by relying on well-trained models to generate high-quality, diverse synthetic data through cross-translation.

Experimental results

Research questions

• RQ1 Can a new data diversification mechanism be designed that goes beyond the current limits of denoising auto-encoding and iterative back-translation in unsupervised MT?
• RQ2 Does cross-model back-translation between two independently trained UMT agents generate more diverse and effective synthetic data than single-model back-translation?
• RQ3 Why does CBD outperform similar variants such as single-model back-translation or ensemble-based data generation in unsupervised settings?
• RQ4 Can CBD achieve state-of-the-art performance without relying on large-scale pretraining or parallel monolingual data?
• RQ5 How does CBD’s data diversity compare to that of standard back-translation and sampling-based augmentation in terms of model performance and intrinsic information capture?

Key findings

• CBD achieves a new state of the art on the WMT’14 English-French unsupervised translation task with 38.2 BLEU, surpassing previous methods.
• CBD sets new SOTA scores on WMT’16 English-German (30.1 BLEU) and English-Romanian (36.3 BLEU), demonstrating broad effectiveness across language pairs.
• CBD improves performance by 1.5–3.3 BLEU on IWSLT’14 English-German and IWSLT’13 English-French benchmarks, indicating strong generalization.
• Extensive ablation studies confirm that CBD’s effectiveness stems from enhanced data diversity, not from model ensembling or direct distillation.
• CBD outperforms alternative methods such as single-model back-translation, sampling-based augmentation, and ensemble distillation, which fail in unsupervised settings due to lack of supervision.
• The method demonstrates that synthetic data generated via cross-model translation is more informative and diverse than that from a single model, leading to better model generalization.