[Paper Review] Word Translation Without Parallel Data

Alexis Conneau, Guillaume Lample+3 more|arXiv (Cornell University)|Oct 11, 2017

Natural Language Processing Techniques35 references251 citations

TL;DR

The paper presents an unsupervised method to learn bilingual word embeddings by aligning monolingual spaces through adversarial training, refined with Procrustes and CSLS, achieving competitive or superior results to supervised methods across several language pairs, including distant and low-resource pairs like English–Esperanto.

ABSTRACT

State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available.

Motivation & Objective

• Develop a method to induce a bilingual dictionary without any parallel data using monolingual corpora only.
• Align monolingual embedding spaces via adversarial training and refine with a Procrustes solution.
• Mitigate hubness and improve translation retrieval using a cross-domain similarity measure (CSLS).
• Provide unsupervised model selection criteria and release resources (dictionaries and embeddings).
• Demonstrate effectiveness on multiple language pairs including distant and low-resource languages.

Proposed method

• Use two monolingual embedding spaces (one per language) and learn a linear mapping W via adversarial training to align them without parallel data.
• Discriminator tries to distinguish mapped source embeddings from target embeddings; the mapping W tries to fool the discriminator (domain-adversarial objective).
• Refine W by constructing a synthetic bilingual dictionary from mutual nearest neighbors and applying the Procrustes solution (orthogonal W) to improve alignment.
• Iteratively generate dictionaries with mutualNNs and CSLS-based matching to further refine W.
• Introduce CSLS (cross-domain similarity local scaling) to reduce hubness by adjusting similarities with neighborhood mean similarities in both domains.
• Apply an orthogonality constraint on W (via an explicit update step) to preserve monolingual embedding properties.
• Propose an unsupervised model selection criterion based on average CSLS-derived translation similarity over the top frequent words as a stopping/hyperparameter choice.

Experimental results

Research questions

• RQ1 Can a high-quality bilingual dictionary be induced without any cross-lingual supervision or parallel data?
• RQ2 How does adversarial alignment compare to supervised methods across diverse language pairs, including distant and non-alphabet-shared languages?
• RQ3 Does the CSLS adjustment mitigate hubness effectively in a bi-domain setting and improve retrieval accuracy?
• RQ4 Can unsupervised model selection reliably indicate mapping quality and guide hyper-parameter choices?
• RQ5 How well does the approach transfer to low-resource language pairs (e.g., English–Esperanto) and to downstream tasks like sentence retrieval and cross-lingual word similarity?

Key findings

• Unsupervised adversarial alignment, followed by Procrustes refinement, achieves competitive or superior word translation accuracy compared with supervised methods on several language pairs.
• CSLS significantly improves word translation retrieval accuracy over standard nearest-neighbor matching, with large gains across languages (e.g., up to ~7.2 percentage points in some pairs).
• The refinement step using synthetic dictionaries from mutual nearest neighbors yields substantial gains, sometimes surpassing the supervised baseline (e.g., English–Italian and English–Spanish on P@1).
• An unsupervised model selection criterion based on average CSLS similarity correlates with actual translation quality and can be used for stopping/hyper-parameter choice.
• The method performs well for distant languages (e.g., English–Russian, English–Chinese) and enables competitive results for low-resource pairs like English–Esperanto, including measurable BLEU gains on simple word-by-word translation setups.