[Paper Review] Refinement of Unsupervised Cross-Lingual Word Embeddings

Magdalena Biesialska, Marta R. Costa‐jussà|arXiv (Cornell University)|Feb 21, 2020

Topic Modeling23 references4 citations

TL;DR

This paper proposes a self-supervised refinement method for unsupervised cross-lingual word embeddings that improves alignment without requiring bilingual supervision. By leveraging a small self-learned seed lexicon and enforcing length- and center-invariance, the approach outperforms state-of-the-art methods in bilingual lexicon induction across multiple language pairs, especially for morphologically rich languages like Finnish and German.

ABSTRACT

Cross-lingual word embeddings aim to bridge the gap between high-resource and low-resource languages by allowing to learn multilingual word representations even without using any direct bilingual signal. The lion's share of the methods are projection-based approaches that map pre-trained embeddings into a shared latent space. These methods are mostly based on the orthogonal transformation, which assumes language vector spaces to be isomorphic. However, this criterion does not necessarily hold, especially for morphologically-rich languages. In this paper, we propose a self-supervised method to refine the alignment of unsupervised bilingual word embeddings. The proposed model moves vectors of words and their corresponding translations closer to each other as well as enforces length- and center-invariance, thus allowing to better align cross-lingual embeddings. The experimental results demonstrate the effectiveness of our approach, as in most cases it outperforms state-of-the-art methods in a bilingual lexicon induction task.

Motivation & Objective

• Address the limitation of orthogonal transformation methods in cross-lingual embedding alignment, which assume isomorphic vector spaces that do not hold for morphologically rich or distant languages.
• Improve unsupervised cross-lingual word embedding quality by refining embeddings without relying on parallel corpora or large bilingual dictionaries.
• Develop a self-supervised refinement framework that uses a small, automatically learned seed lexicon to guide alignment in a shared vector space.
• Enhance embedding robustness by enforcing length-invariance and center-invariance, mitigating issues from imperfect isomorphism in language vector spaces.
• Demonstrate consistent improvements over state-of-the-art unsupervised methods on standard bilingual lexicon induction benchmarks.

Proposed method

• Propose a self-supervised refinement framework that learns from a small, automatically extracted seed lexicon derived from monolingual corpora.
• Apply a linear transformation to align source and target language embeddings in a shared space, with constraints enforcing unit length and zero-centered vectors.
• Introduce a loss function that minimizes the distance between word vectors and their corresponding translations in the shared space.
• Enforce length-invariance by normalizing all vectors to unit length and center-invariance by ensuring the centroid of the embedding space is at the origin.
• Optimize the transformation matrix using gradient descent on the refined objective, improving alignment without explicit bilingual supervision.
• Use the CSLS (Centroid-based Similarity Learning) method for nearest neighbor retrieval during evaluation, ensuring fair comparison with baselines.

Experimental results

Research questions

• RQ1 Can a self-supervised refinement method improve unsupervised cross-lingual word embeddings without requiring any parallel or bilingual data?
• RQ2 How effective is the proposed method in handling non-isomorphic language pairs, particularly morphologically rich languages like Finnish and German?
• RQ3 Does enforcing length- and center-invariance in the shared embedding space lead to better alignment than standard orthogonal transformation?
• RQ4 How does the performance of the proposed method compare to state-of-the-art unsupervised and weakly supervised CLE models on standard benchmark tasks?
• RQ5 Can a small, self-learned seed lexicon be sufficient to guide effective refinement of unsupervised CLEs?

Key findings

• The proposed method outperforms the unsupervised VecMap baseline in bilingual lexicon induction for English-Spanish and English-Finnish, achieving higher P@1 scores.
• For the English-German pair, the method performs on par with the baseline, demonstrating robustness even in challenging morphologically rich language settings.
• The method surpasses the IterNorm refinement technique—originally designed for supervised settings—when applied to unsupervised CLEs, indicating superior generalization in low-resource scenarios.
• The use of a self-learned seed lexicon significantly enhances alignment quality, suggesting that even small, automatically extracted bilingual signals can guide effective refinement.
• The combination of length- and center-invariance constraints leads to more stable and accurate cross-lingual alignments, especially when isomorphism assumptions fail.
• The model achieves state-of-the-art performance on the benchmark dataset across nearly all evaluated language pairs, confirming the effectiveness of the self-supervised refinement strategy.