[Paper Review] MuRIL: Multilingual Representations for Indian Languages

Simran Khanuja, Diksha Bansal+12 more|arXiv (Cornell University)|Mar 19, 2021

Natural Language Processing Techniques14 references158 citations

TL;DR

MuRIL is a multilingual BERT-style encoder built for Indian languages, trained with monolingual MLM data plus translated and transliterated cross-lingual data, and it outperforms mBERT on XTREME IN benchmarks especially on transliterated sets.

ABSTRACT

India is a multilingual society with 1369 rationalized languages and dialects being spoken across the country (INDIA, 2011). Of these, the 22 scheduled languages have a staggering total of 1.17 billion speakers and 121 languages have more than 10,000 speakers (INDIA, 2011). India also has the second largest (and an ever growing) digital footprint (Statista, 2020). Despite this, today's state-of-the-art multilingual systems perform suboptimally on Indian (IN) languages. This can be explained by the fact that multilingual language models (LMs) are often trained on 100+ languages together, leading to a small representation of IN languages in their vocabulary and training data. Multilingual LMs are substantially less effective in resource-lean scenarios (Wu and Dredze, 2020; Lauscher et al., 2020), as limited data doesn't help capture the various nuances of a language. One also commonly observes IN language text transliterated to Latin or code-mixed with English, especially in informal settings (for example, on social media platforms) (Rijhwani et al., 2017). This phenomenon is not adequately handled by current state-of-the-art multilingual LMs. To address the aforementioned gaps, we propose MuRIL, a multilingual LM specifically built for IN languages. MuRIL is trained on significantly large amounts of IN text corpora only. We explicitly augment monolingual text corpora with both translated and transliterated document pairs, that serve as supervised cross-lingual signals in training. MuRIL significantly outperforms multilingual BERT (mBERT) on all tasks in the challenging cross-lingual XTREME benchmark (Hu et al., 2020). We also present results on transliterated (native to Latin script) test sets of the chosen datasets and demonstrate the efficacy of MuRIL in handling transliterated data.

Motivation & Objective

• Motivate the need for language models tailored to Indian languages due to limited IN-language representation in existing multilingual SMs.
• Develop MuRIL, a monolingual-text–augmented multilingual LM with MLM and supervised Translation Language Modeling (TLM).
• Evaluate MuRIL against mBERT on cross-lingual benchmarks (XTREME) and transliterated test sets to demonstrate robustness in IN languages.

Proposed method

• Train a BERT-base encoder with MLM on monolingual IN data (17 languages plus English).
• Incorporate supervised cross-lingual signals via Translation Language Modeling (TLM) using translated and transliterated parallel data.
• Construct a focused IN-language WordPiece vocabulary of 197,285 tokens built from upsampled, language-smoothed corpora.
• Upsample monolingual data (Wikipedia) to balance language representation.
• Pre-train on ~16B tokens, then fine-tune/evaluate in zero-shot setting on XTREME IN tasks.
• Release MuRIL encoder and pre-trained models for TFHub and HuggingFace.

Experimental results

Research questions

• RQ1 How does a language-model pre-trained specifically on Indian languages perform on cross-lingual transfer tasks compared to general multilingual LMs like mBERT?
• RQ2 Does incorporating translated and transliterated cross-lingual data improve performance on transliterated test sets in IN languages?
• RQ3 What is the impact of a language-focused vocabulary on model effectiveness for IN languages?
• RQ4 How well does MuRIL fare on XTREME IN benchmarks in zero-shot transfer from English training data?
• RQ5 Can MuRIL handle transliterated data more effectively than mBERT?

Key findings

• MuRIL significantly outperforms mBERT across XTREME IN tasks (PANX, UDPOS, XNLI, Tatoeba) with an average score of 68.6 vs 59.1 for mBERT on Table 1.
• MuRIL also shows substantially higher performance on transliterated test sets than mBERT (Table 2 results show gains across transliterated evaluations).
• Across IN languages, MuRIL attains higher per-language scores in PANX, UDPOS, XNLI, and XQuAD/MLQA-style evaluations, demonstrating robustness to transliteration and script variation.
• MuRIL’s dedicated IN-language vocabulary and transliteration-aware training contribute to lower fertility in tokenization, preserving semantic content better than mBERT.
• The model is released publicly (TFHub and HuggingFace) to enable broader use in Indian-language NLP applications.