[Paper Review] Explaining and Generalizing Back-Translation through Wake-Sleep

Ryan Cotterell, Julia Kreutzer|arXiv (Cornell University)|Jun 12, 2018

Topic Modeling28 references16 citations

TL;DR

This paper provides a principled variational inference interpretation of back-translation as a single iteration of the wake-sleep algorithm in a generative model of bitext. It proposes an iterative generalization of back-translation that improves translation performance by up to 1.6 BLEU on English–German and English–Latvian translation tasks, demonstrating consistent gains in both in-domain and cross-domain settings.

ABSTRACT

Back-translation has become a commonly employed heuristic for semi-supervised neural machine translation. The technique is both straightforward to apply and has led to state-of-the-art results. In this work, we offer a principled interpretation of back-translation as approximate inference in a generative model of bitext and show how the standard implementation of back-translation corresponds to a single iteration of the wake-sleep algorithm in our proposed model. Moreover, this interpretation suggests a natural iterative generalization, which we demonstrate leads to further improvement of up to 1.6 BLEU.

Motivation & Objective

• To provide a principled generative model interpretation of back-translation in neural machine translation.
• To show that standard back-translation corresponds to a single wake-sleep iteration in a variational inference framework.
• To propose and evaluate an iterative generalization of back-translation based on the wake-sleep algorithm.
• To demonstrate consistent improvements in translation quality across multiple language pairs and domains.
• To establish a theoretical foundation for back-translation that enables future methodological extensions.

Proposed method

• The authors construct a fully generative model of bitext where the joint distribution is factorized as p(y|x) * p(x), with p(y|x) as the translation model and p(x) as the language model.
• Back-translation is interpreted as a variational approximation where the backward translator acts as an inference network for the latent source sentence x given a target sentence y.
• The method frames back-translation as one iteration of the wake-sleep algorithm, with the 'wake' phase optimizing the translation model and the 'sleep' phase refining the inference network.
• The proposed generalization extends this to multiple wake-sleep iterations, iteratively re-estimating both forward and backward translation models.
• The model is trained using maximum likelihood on bitext and fine-tuned via back-translation on monolingual data, with early stopping on development sets.
• Experiments use a standard sequence-to-sequence model with attention, GRUs, subword tokenization, and standard regularization techniques.

Experimental results

Research questions

• RQ1 How can back-translation be formally interpreted within a generative modeling framework?
• RQ2 What is the relationship between back-translation and the wake-sleep algorithm in variational inference?
• RQ3 Can iterative refinement of both forward and backward translation models lead to improved translation performance?
• RQ4 Does the proposed iterative generalization of back-translation yield consistent gains across different language pairs and domains?
• RQ5 Can a principled interpretation of back-translation guide future improvements in semi-supervised neural machine translation?

Key findings

• The proposed iterative generalization of back-translation improves translation performance by up to 1.6 BLEU over standard back-translation on English–German and English–Latvian translation tasks.
• The largest improvements occur in domain adaptation settings, such as using TED talks data for semi-supervised adaptation, where gains are most pronounced.
• Improvements are consistent across both in-domain (WMT) and cross-domain (TED) settings, with the first iteration of back-translation yielding the largest relative gains in most cases.
• The method demonstrates that back-translation is equivalent to a single wake-sleep iteration, validating its connection to variational inference in a generative model.
• The iterative refinement process leads to better alignment of monolingual data with the target distribution, reducing noise and improving model generalization.
• The results suggest that principled generative modeling can yield practical improvements in semi-supervised neural machine translation.