[Paper Review] Learning Contextualised Cross-lingual Word Embeddings for Extremely Low-Resource Languages Using Parallel Corpora.

Takashi Wada, Tomoharu Iwata+3 more|arXiv (Cornell University)|Oct 27, 2020

Topic Modeling55 references4 citations

TL;DR

This paper proposes a low-resource cross-lingual word embedding method using a shared LSTM-based encoder-decoder model trained on small parallel corpora (e.g., a few hundred sentence pairs). By jointly learning contextualized embeddings across languages through bidirectional translation and reconstruction, and combining word and subword representations, the model achieves state-of-the-art performance in bilingual lexicon induction and word alignment for endangered languages like Yongning Na, Shipibo-Konibo, and Griko.

ABSTRACT

We propose a new approach for learning contextualised cross-lingual word embeddings based only on a small parallel corpus (e.g. a few hundred sentence pairs). Our method obtains word embeddings via an LSTM-based encoder-decoder model that performs bidirectional translation and reconstruction of the input sentence. Through sharing model parameters among different languages, our model jointly trains the word embeddings in a common multilingual space. We also propose a simple method to combine word and subword embeddings to make use of orthographic similarities across different languages. We base our experiments on real-world data from endangered languages, namely Yongning Na, Shipibo-Konibo and Griko. Our experiments on bilingual lexicon induction and word alignment tasks show that our model outperforms existing methods by a large margin for most language pairs. These results demonstrate that, contrary to common belief, an encoder-decoder translation model is beneficial for learning cross-lingual representations, even in extremely low-resource scenarios.

Motivation & Objective

• Address the challenge of learning effective cross-lingual representations for extremely low-resource and endangered languages with minimal parallel data.
• Overcome the limitations of existing methods that struggle with data scarcity by leveraging a shared multilingual architecture.
• Improve cross-lingual transfer performance in low-resource settings through joint training of word embeddings across languages.
• Utilize orthographic similarities across languages by combining word and subword embeddings to enhance representation quality.

Proposed method

• Train an LSTM-based encoder-decoder model to perform bidirectional translation and sentence reconstruction on small parallel corpora.
• Share model parameters across languages to jointly learn word embeddings in a shared multilingual embedding space.
• Use a reconstruction loss to preserve semantic and syntactic structure during translation, enhancing embedding quality.
• Combine word-level and subword-level embeddings using a learnable mechanism to exploit cross-linguistic orthographic similarities.
• Train the model end-to-end using only parallel sentence pairs, without requiring monolingual data or external resources.
• Apply the learned embeddings to downstream tasks such as bilingual lexicon induction and word alignment.

Experimental results

Research questions

• RQ1 Can an encoder-decoder model trained on a few hundred parallel sentence pairs effectively learn contextualized cross-lingual word embeddings in extremely low-resource settings?
• RQ2 Does parameter sharing across languages improve the quality of cross-lingual representations compared to monolingual or isolated training?
• RQ3 To what extent do combined word and subword embeddings enhance performance on cross-lingual tasks for morphologically diverse or low-resource languages?
• RQ4 How does the proposed method compare to existing state-of-the-art approaches in bilingual lexicon induction and word alignment for endangered languages?

Key findings

• The proposed model outperforms existing methods by a large margin on bilingual lexicon induction and word alignment tasks across multiple language pairs.
• The model achieves strong performance even with only a few hundred parallel sentence pairs, demonstrating effectiveness in extremely low-resource scenarios.
• Joint training via bidirectional translation and reconstruction significantly improves the quality of cross-lingual word embeddings.
• The combination of word and subword embeddings leads to better capture of cross-linguistic similarities, especially in languages with shared orthographic features.
• The method achieves state-of-the-art results on real-world data from endangered languages, including Yongning Na, Shipibo-Konibo, and Griko.
• The results challenge the common belief that sequence-to-sequence models are unsuitable for cross-lingual representation learning in low-resource settings.