[Paper Review] Unsupervised Cross-lingual Transfer of Word Embedding Spaces

Ruochen Xu, Yiming Yang+2 more|arXiv (Cornell University)|Sep 10, 2018

Topic Modeling4 citations

TL;DR

This paper proposes an unsupervised method for cross-lingual word embedding transfer that jointly optimizes bidirectional mappings between monolingual embedding spaces using Sinkhorn distance for distributional matching and back-translation loss. The approach achieves state-of-the-art performance on bilingual lexicon induction and cross-lingual similarity tasks without any parallel data or bilingual lexicons, demonstrating strong robustness across diverse language pairs.

ABSTRACT

Cross-lingual transfer of word embeddings aims to establish the semantic mappings among words in different languages by learning the transformation functions over the corresponding word embedding spaces. Successfully solving this problem would benefit many downstream tasks such as to translate text classification models from resource-rich languages (e.g. English) to low-resource languages. Supervised methods for this problem rely on the availability of cross-lingual supervision, either using parallel corpora or bilingual lexicons as the labeled data for training, which may not be available for many low resource languages. This paper proposes an unsupervised learning approach that does not require any cross-lingual labeled data. Given two monolingual word embedding spaces for any language pair, our algorithm optimizes the transformation functions in both directions simultaneously based on distributional matching as well as minimizing the back-translation losses. We use a neural network implementation to calculate the Sinkhorn distance, a well-defined distributional similarity measure, and optimize our objective through back-propagation. Our evaluation on benchmark datasets for bilingual lexicon induction and cross-lingual word similarity prediction shows stronger or competitive performance of the proposed method compared to other state-of-the-art supervised and unsupervised baseline methods over many language pairs.

Motivation & Objective

• To address the challenge of cross-lingual word embedding alignment in low-resource languages where parallel corpora or bilingual lexicons are unavailable.
• To develop a method that learns semantic mappings between monolingual word embeddings without any cross-lingual supervision.
• To improve robustness and performance over existing unsupervised and supervised baselines on cross-lingual transfer benchmarks.
• To jointly optimize forward and reverse translation mappings to enforce consistency and reduce error accumulation.

Proposed method

• The method uses a neural network to compute the Sinkhorn distance—a differentiable, well-defined measure of distributional similarity—between transformed word embeddings in both directions.
• It jointly optimizes two objectives: distributional matching via Sinkhorn distance and back-translation loss to minimize reconstruction error after two-step translation.
• The model is trained end-to-end via back-propagation, with the Sinkhorn distance implemented using a differentiable neural layer that solves the optimal transport problem approximately.
• The framework is symmetric: it learns transformations from language A to B and from B to A simultaneously, ensuring consistency in both directions.
• The method does not require any parallel sentences, bilingual dictionaries, or human-annotated supervision, relying solely on monolingual word embeddings.
• It uses cosine similarity between transformed embeddings and target word vectors as the similarity score for downstream evaluation.

Experimental results

Research questions

• RQ1 Can a fully unsupervised method achieve competitive performance in cross-lingual word embedding transfer without any parallel data or bilingual lexicons?
• RQ2 How does bidirectional optimization using Sinkhorn distance compare to unidirectional or nearest-neighbor-based methods in terms of robustness and accuracy?
• RQ3 Does the use of back-translation loss improve the quality of learned mappings compared to distributional matching alone?
• RQ4 How does the method perform across diverse language pairs, especially those with low-resource characteristics?
• RQ5 Can the model generalize well to both easy and hard language pairs, as measured by standard benchmarks?

Key findings

• The proposed method achieves state-of-the-art performance on the LEX-Z benchmark for bilingual lexicon induction, outperforming other unsupervised methods and matching or exceeding supervised baselines on several language pairs.
• On the LEX-C dataset, which features higher-quality supervision and larger candidate sets, the method still performs strongly, especially on harder language pairs like de-en and fa-en.
• The model achieves a Pearson correlation of 0.71 on the cross-lingual word similarity task (Conneau et al., 2017), matching the best supervised methods and outperforming other unsupervised models.
• The method shows robust performance across language pairs and translation directions, with consistent results even when the source language has more unique words than the target.
• The use of Sinkhorn distance leads to more stable and higher-quality mappings than KL-divergence or nearest-neighbor matching, as validated by ablation and comparative analysis.
• The bidirectional training scheme significantly improves robustness and reduces error accumulation, as evidenced by the low back-translation loss and high consistency in reconstructed embeddings.