[Paper Review] WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

Benjamin Minixhofer, Fabian Paischer+1 more|arXiv (Cornell University)|Dec 13, 2021

Topic ModelingComputer Science47 references25 citations

TL;DR

WECHSEL is a method that transfers monolingual language models to new languages by initializing subword embeddings using multilingual static word embeddings, achieving performance comparable to models trained from scratch with up to 64x less training effort. It outperforms random initialization and prior transfer methods like TransInner across multiple languages, including low-resource ones.

ABSTRACT

Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method -- called WECHSEL -- to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.

Motivation & Objective

• To address the high computational and environmental cost of training large language models from scratch in non-English languages.
• To improve cross-lingual transfer efficiency by leveraging multilingual static word embeddings for subword embedding initialization.
• To enable effective transfer of monolingual models like RoBERTa and GPT-2 to low- and medium-resource languages with minimal training.
• To reduce reliance on massive multilingual models that suffer from performance degradation due to the 'curse of multilinguality'.
• To make training large language models in new languages more accessible and environmentally sustainable.

Proposed method

• Transfer monolingual language models by copying all non-embedding parameters from a source English model to a target language model.
• Replace the English tokenizer with a target language tokenizer to support subword tokenization in the new language.
• Initialize subword embeddings in the target language by mapping multilingual static word embeddings to subword units using semantic similarity.
• Use multilingual word embeddings (e.g., from fastText) to align target language subwords with semantically similar English subwords.
• Train the transferred model with minimal fine-tuning steps, significantly reducing training cost compared to training from scratch.
• Apply the method to both encoder (RoBERTa) and decoder (GPT-2) architectures across multiple languages, including low-resource ones.

Experimental results

Research questions

• RQ1 Can subword embeddings be effectively initialized using multilingual static word embeddings to improve cross-lingual transfer of monolingual language models?
• RQ2 Does WECHSEL reduce the number of training steps required to achieve high performance compared to random initialization or prior transfer methods?
• RQ3 How effective is WECHSEL on low-resource languages where data and compute are limited?
• RQ4 Does the method outperform models of comparable size that were trained from scratch with significantly more compute?
• RQ5 Is freezing non-embedding parameters necessary when using WECHSEL, compared to methods like TransInner?

Key findings

• WECHSEL outperforms randomly initialized models (FullRand) and the TransInner method across all languages and tasks, including NER and NLI for RoBERTa and perplexity for GPT-2.
• For RoBERTa, WECHSEL achieved state-of-the-art performance on French, German, Chinese, and Swahili NER and NLI tasks with only 1/64th of the training steps required by models like CamemBERT and GBERTBase.
• For GPT-2, WECHSEL achieved lower perplexity than both FullRand and TransInner on medium- and low-resource languages, with consistent improvements even in low-data regimes.
• On low-resource languages like Sundanese, Scottish Gaelic, Uyghur, and Malagasy, WECHSEL showed stronger performance gains as data scarcity increased, indicating higher robustness to low-resource settings.
• Freezing non-embedding parameters was not necessary when using WECHSEL, unlike with TransInner, suggesting that the semantic initialization stabilizes training from the start.
• The method enables training effective monolingual language models in new languages with up to 64x less training effort than comparable models trained from scratch.