[Paper Review] What the [MASK]? Making Sense of Language-Specific BERT Models

Debora Nozza, Federico Bianchi+1 more|arXiv (Cornell University)|Mar 5, 2020

Topic ModelingComputer Science31 references85 citations

TL;DR

This paper surveys language-specific BERT models, compares them to mBERT, and introduces BertLang, an interactive site to explore language-specific BERT performance across languages, tasks, and domains.

ABSTRACT

Recently, Natural Language Processing (NLP) has witnessed an impressive progress in many areas, due to the advent of novel, pretrained contextual representation models. In particular, Devlin et al. (2019) proposed a model, called BERT (Bidirectional Encoder Representations from Transformers), which enables researchers to obtain state-of-the art performance on numerous NLP tasks by fine-tuning the representations on their data set and task, without the need for developing and training highly-specific architectures. The authors also released multilingual BERT (mBERT), a model trained on a corpus of 104 languages, which can serve as a universal language model. This model obtained impressive results on a zero-shot cross-lingual natural inference task. Driven by the potential of BERT models, the NLP community has started to investigate and generate an abundant number of BERT models that are trained on a particular language, and tested on a specific data domain and task. This allows us to evaluate the true potential of mBERT as a universal language model, by comparing it to the performance of these more specific models. This paper presents the current state of the art in language-specific BERT models, providing an overall picture with respect to different dimensions (i.e. architectures, data domains, and tasks). Our aim is to provide an immediate and straightforward overview of the commonalities and differences between Language-Specific (language-specific) BERT models and mBERT. We also provide an interactive and constantly updated website that can be used to explore the information we have collected, at https://bertlang.unibocconi.it.

Motivation & Objective

• Provide an overview of language-specific BERT models and how they differ from mBERT.
• Summarize architectures, data domains, and tasks where language-specific BERT models are evaluated.
• Present an interactive resource (BertLang) to compare models and guide model selection.
• Highlight key findings on performance advantages and variability across languages and tasks.

Proposed method

• Review existing language-specific BERT models across 18 languages and 29 NLP tasks.
• Aggregate and compare performance with mBERT where available.
• Introduce and describe the BertLang website as an interactive, up-to-date repository.
• Discuss data sources used for pretraining (Wikipedia, OPUS, OSCAR) and evaluation domains (news, social media, etc.).
• Summarize observed trends such as when language-specific models outperform mBERT and in which tasks.

Experimental results

Research questions

• RQ1 How do language-specific BERT models compare to mBERT across languages and tasks?
• RQ2 Which architectures (BERT, ALBERT, RoBERTa) and data sources yield the best performance for language-specific models?
• RQ3 What domains and datasets influence the relative gains of language-specific models over mBERT?
• RQ4 How widely available are language-specific models across languages, and how can researchers efficiently find suitable models?

Key findings

• On average, language-specific BERT models outperform mBERT across the considered tasks.
• Low-resource languages (e.g., Yorùbá, Arabic) show the largest improvements relative to mBERT.
• Named Entity Recognition is the most frequently reported task among language-specific models in the dataset.
• Sentiment analysis often exhibits substantial gains with language-specific models.
• Performance gains vary markedly by task and language, with some tasks (e.g., noun sense disambiguation) favoring mBERT in certain cases.
• The paper introduces BertLang as a central, up-to-date resource for exploring language-specific BERT models.