[Paper Review] A Primer on Pretrained Multilingual Language Models

Sumanth Doddapaneni, G. Ramesh+3 more|arXiv (Cornell University)|Jul 1, 2021

Domain Adaptation and Few-Shot Learning43 citations

TL;DR

A survey of multilingual language models (MLLMs) covering their architectures, objectives, data, benchmarks, and their comparative performance with monolingual models, along with future directions.

ABSTRACT

Multilingual Language Models (\MLLMs) such as mBERT, XLM, XLM-R, extit{etc.} have emerged as a viable option for bringing the power of pretraining to a large number of languages. Given their success in zero-shot transfer learning, there has emerged a large body of work in (i) building bigger \MLLMs~covering a large number of languages (ii) creating exhaustive benchmarks covering a wider variety of tasks and languages for evaluating \MLLMs~ (iii) analysing the performance of \MLLMs~on monolingual, zero-shot cross-lingual and bilingual tasks (iv) understanding the universal language patterns (if any) learnt by \MLLMs~ and (v) augmenting the (often) limited capacity of \MLLMs~ to improve their performance on seen or even unseen languages. In this survey, we review the existing literature covering the above broad areas of research pertaining to \MLLMs. Based on our survey, we recommend some promising directions of future research.

Motivation & Objective

• Review how MLLMs are built and how they differ across models.
• Summarize training objectives (monolingual and parallel) and data sources used for pretraining.
• Survey benchmarks used to evaluate MLLMs across languages and tasks.
• Discuss whether MLLMs outperform monolingual LMs for specific languages and tasks.
• Highlight methods to extend MLLMs to unseen languages and recommend future directions.

Proposed method

• Catalog architectures of representative MLLMs and their configurations (N, k, d, parameters).
• Compare pretraining objectives: MLM, CLM, MRTD, TLM, CAMLM, CLMLM, XLCO, HICTL, CLSA and related methods.
• Describe monolingual vs. parallel objective combinations and their data requirements.
• Explain pretraining data choices and language sampling via exponential smoothing to balance language exposure.
• Summarize evaluation benchmarks like XGLUE, XTREME, XTREME-R and XGLUE across tasks.
• Synthesize findings on cross-lingual transfer, bilingual tasks, and universal-pattern hypotheses.

Experimental results

Research questions

• RQ1 How are different MLLMs built and how do they differ from each other?
• RQ2 What benchmarks are used to evaluate MLLMs?
• RQ3 Are MLLMs better than monolingual LMs for a given language?
• RQ4 Do MLLMs enable zero-shot cross-lingual transfer and bilingual tasks?
• RQ5 Do MLLMs reveal universal language patterns and how to extend them to new languages?

Key findings

• MLLMs vary in architecture, pretraining data, languages covered, and vocabulary size; higher capacity models generally perform better in benchmarks.
• Cross-lingual transfer ability depends on factors like shared vocabulary, representation alignment, and data size; no single model outperforms monolingual LMs consistently across all settings.
• Benchmarks such as XGLUE, XTREME, and XTREME-R are used to assess cross-lingual performance across classification, structure prediction, QA, and retrieval tasks.
• There is evidence that zero-shot cross-lingual transfer and bilingual tasks benefit from MLLMs, especially in low-resource languages, but universal interlingua patterns are not yet established.
• Extending MLLMs to unseen languages can be approached via fine-tuning, adapters, and other extension methods.