[Paper Review] Unsupervised Hyperalignment for Multilingual Word Embeddings

Jean Alaux, Édouard Grave+2 more|arXiv (Cornell University)|Nov 2, 2018

Topic Modeling39 references41 citations

TL;DR

This paper extends unsupervised bilingual word embedding alignment to a multilingual setting by learning composable mappings to a common pivot space, enforcing transitive translations and improving indirect word translation among eleven languages.

ABSTRACT

We consider the problem of aligning continuous word representations, learned in multiple languages, to a common space. It was recently shown that, in the case of two languages, it is possible to learn such a mapping without supervision. This paper extends this line of work to the problem of aligning multiple languages to a common space. A solution is to independently map all languages to a pivot language. Unfortunately, this degrades the quality of indirect word translation. We thus propose a novel formulation that ensures composable mappings, leading to better alignments. We evaluate our method by jointly aligning word vectors in eleven languages, showing consistent improvement with indirect mappings while maintaining competitive performance on direct word translation.

Motivation & Objective

• Motivate multilingual alignment of word embeddings without supervision.
• Extend bilingual unsupervised alignment to multiple languages with composable mappings.
• Enforce transitivity of translations by aligning all languages to a common pivot space.
• Evaluate on eleven languages to assess direct and indirect translation quality.

Proposed method

• Define a common pivot space where each language is mapped by an orthogonal matrix Q_i and aligned via a permutation P_ij.
• Use a multilingual objective that minimizes the sum of RCSLS-based losses across language pairs, weighted by α_ij, to encourage coherent alignments in the pivot space.
• Adopt a two-stage optimization: first train with L2 loss and Sinkhorn-based assignment, then switch to a faster, approximate RCSLS loss for efficiency.
• Initialize with a Gromov-Wasserstein-based scheme to obtain a good starting permutation P.
• Employ stochastic sampling of language pairs to scale the optimization to multiple languages.
• Evaluate both direct translations to/from the pivot and indirect translations between non-pivot languages.

Experimental results

Research questions

• RQ1 Can multiple languages be aligned to a single common space in an unsupervised manner while preserving or improving indirect translations between non-pivot language pairs?
• RQ2 Does enforcing compositionality via a common pivot space reduce degradation of indirect translations compared to independent bilingual alignments?
• RQ3 How do weighting schemes α_ij and initialization affect multilingual alignment quality and scalability?
• RQ4 What is the trade-off between direct and indirect translation performance when extending from bilingual to multilingual unsupervised alignment?

Key findings

• UMH achieves competitive direct word translation performance relative to bilingual and other multilingual approaches.
• Indirect translation quality improves significantly when enforcing composable multilingual alignments, particularly for distant language pairs.
• A multilingual setup with 11 languages preserves strong direct translations while yielding robust indirect translations across language families.
• Weighting schemes favoring direct pivot connections help maintain pivot translations as the number of languages grows without severely sacrificing indirect translations.
• Initialization via Gromov-Wasserstein, combined with staged optimization, provides practical convergence and scalable training on CPU.