[Paper Review] Unsupervised Machine Translation Using Monolingual Corpora Only

Guillaume Lample, Alexis Conneau+2 more|arXiv (Cornell University)|Oct 31, 2017

Natural Language Processing Techniques36 references198 citations

TL;DR

The paper presents an unsupervised neural machine translation approach that learns to translate using only monolingual data by aligning two languages in a shared latent space, via denoising auto-encoding, cross-domain translation, and adversarial training, initialized from a word-by-word unsupervised dictionary.

ABSTRACT

Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences. In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data. We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space. By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data. We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.

Motivation & Objective

• Motivate translation in zero-parallel-data scenarios and establish a strong lower bound for semi-supervised methods.
• Develop an end-to-end unsupervised MT model that maps two languages into a common latent space.
• Leverage denoising auto-encoding, cross-domain translation, and adversarial distribution alignment to learn translation.
• Provide an iterative training procedure that improves translation quality without labeled data.

Proposed method

• Use a single encoder/decoder for both languages with language-specific lookup tables.
• Train with a combination of denoising auto-encoding losses in both languages.
• Incorporate a cross-domain translation objective by translating across languages using the current model and reconstructing the original sentence.
• Employ an adversarial loss to align the latent representations of the two languages via a discriminator that predicts language from encodings and is fooled by the encoder.
• Initialize from an unsupervised word-by-word translation dictionary learned from monolingual data and iteratively improve through back-translation.
• Optimize the full objective as a weighted sum of auto-encoding, cross-domain, and adversarial losses; update discriminator jointly.
• Use iterative training (M^(t) -> M^(t+1)) to progressively improve translation quality.

Experimental results

Research questions

• RQ1 Can machine translation be learned without any parallel data given monolingual corpora in two languages?
• RQ2 Does mapping both languages to a shared latent space enable effective cross-language decoding?
• RQ3 How do denoising, cross-domain reconstruction, and adversarial alignment contribute to unsupervised translation quality?
• RQ4 What is the impact of iterative training and initialization from monolingual resources on BLEU scores?
• RQ5 How can one select hyper-parameters without parallel data?

Key findings

• On Multi30k-Task1 English-French, BLEU improves from 27.48 (iteration 1) to 32.76 (iteration 3).
• On Multi30k-Task1 English-German, BLEU improves from 23.69 (iteration 1) to 26.26 (iteration 3).
• On WMT English-French, BLEU improves from 12.10 (iteration 1) to 15.05 (iteration 3).
• On WMT French-English, BLEU improves from 11.79 (iteration 1) to 14.31 (iteration 3).
• The unsupervised model, with monolingual data in both languages, achieves performance comparable to a supervised MT system trained on about 100,000 parallel sentences for the WMT en-fr pair.
• Discriminator-based adversarial alignment and the combination of auto-encoding and cross-domain losses are critical for performance, as shown by ablation results.