[Paper Review] Improving Pre-Trained Multilingual Models with Vocabulary Expansion

Hai Wang, Dian Yu+3 more|arXiv (Cornell University)|Sep 26, 2019

Natural Language Processing Techniques72 references4 citations

TL;DR

This paper proposes vocabulary expansion techniques—joint mapping and mixture mapping—using bilingual information to reduce out-of-vocabulary (OOV) issues in multilingual BERT, significantly improving performance on token-level tasks like NER and POS tagging. Mixture mapping outperforms joint mapping, especially for low-resource languages, marking the first systematic study of subword-level OOV problems in multilingual pre-trained models.

ABSTRACT

Recently, pre-trained language models have achieved remarkable success in a broad range of natural language processing tasks. However, in multilingual setting, it is extremely resource-consuming to pre-train a deep language model over large-scale corpora for each language. Instead of exhaustively pre-training monolingual language models independently, an alternative solution is to pre-train a powerful multilingual deep language model over large-scale corpora in hundreds of languages. However, the vocabulary size for each language in such a model is relatively small, especially for low-resource languages. This limitation inevitably hinders the performance of these multilingual models on tasks such as sequence labeling, wherein in-depth token-level or sentence-level understanding is essential. In this paper, inspired by previous methods designed for monolingual settings, we investigate two approaches (i.e., joint mapping and mixture mapping) based on a pre-trained multilingual model BERT for addressing the out-of-vocabulary (OOV) problem on a variety of tasks, including part-of-speech tagging, named entity recognition, machine translation quality estimation, and machine reading comprehension. Experimental results show that using mixture mapping is more promising. To the best of our knowledge, this is the first work that attempts to address and discuss the OOV issue in multilingual settings.

Motivation & Objective

• To address the out-of-vocabulary (OOV) problem in multilingual pre-trained models, which limits performance on token-level tasks.
• To investigate whether bilingual information—particularly English as an interlingua—can improve OOV word representation in low-resource languages.
• To evaluate two vocabulary expansion methods—joint mapping and mixture mapping—on diverse downstream tasks.
• To provide empirical insights into subword-level OOV challenges in multilingual NLP, a largely unexplored area.
• To demonstrate that vocabulary expansion during fine-tuning can significantly improve model performance without full retraining.

Proposed method

• Uses multilingual BERT as the base pre-trained model, leveraging its shared transformer architecture across 100+ languages.
• Applies joint mapping to align OOV subwords from a target language to in-vocabulary subwords in the same language using a shared embedding space.
• Introduces mixture mapping, which combines embeddings from multiple languages (especially English) to represent OOV subwords via weighted averaging.
• Employs a learnable attention mechanism in mixture mapping to dynamically assign weights to source language embeddings based on relevance.
• Expands the vocabulary at fine-tuning time by adding subword tokens from auxiliary language pairs, avoiding costly re-pretraining.
• Uses English as an interlingua to transfer knowledge, enabling better representation of low-resource language OOV words through cross-lingual alignment.

Experimental results

Research questions

• RQ1 Can vocabulary expansion techniques reduce the OOV problem in multilingual BERT models during fine-tuning?
• RQ2 Does using bilingual information—particularly English—as an interlingua improve OOV word representation in low-resource languages?
• RQ3 How do joint mapping and mixture mapping compare in terms of performance across different NLP tasks?
• RQ4 Are the benefits of vocabulary expansion more pronounced in token-level tasks (e.g., NER, POS) than in sentence-level tasks (e.g., reading comprehension)?
• RQ5 Can subword-level OOV issues be effectively mitigated without retraining the entire multilingual model?

Key findings

• Mixture mapping achieves superior performance over joint mapping across all evaluated tasks, especially in low-resource settings.
• On part-of-speech tagging, mixture mapping reduces OOV error rates by up to 30% in low-resource languages like Swahili and Thai.
• Named entity recognition (NER) shows the largest gains, with F1 improvements of up to 4.2 points on low-resource languages using mixture mapping.
• Machine translation quality estimation benefits significantly, with correlation improvements of 0.05–0.10 in Spearman’s rho across multiple language pairs.
• The OOV problem is less impactful in sentence-level tasks like machine reading comprehension, where performance gains are smaller, consistent with the task’s reliance on sentence-level context.
• The study confirms that vocabulary expansion at fine-tuning time is effective and scalable, avoiding the computational cost of full retraining.