[Paper Review] MarIA: Spanish Language Models

Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé+8 more|arXiv (Cornell University)|Jul 15, 2021

Natural Language Processing Techniques58 citations

TL;DR

The paper presents four large Spanish language models (RoBERTa-base, RoBERTa-large, GPT-2 variants) trained on a huge Spanish pretraining corpus from the National Library of Spain, plus a new QA dataset SQAC, and evaluates these models against multilingual and Spanish baselines across 9 tasks.

ABSTRACT

This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community. Currently, MarIA includes RoBERTa-base, RoBERTa-large, GPT2 and GPT2-large Spanish language models, which can arguably be presented as the largest and most proficient language models in Spanish. The models were pretrained using a massive corpus of 570GB of clean and deduplicated texts with 135 billion words extracted from the Spanish Web Archive crawled by the National Library of Spain between 2009 and 2019. We assessed the performance of the models with nine existing evaluation datasets and with a novel extractive Question Answering dataset created ex novo. Overall, MarIA models outperform the existing Spanish models across a variety of NLU tasks and training settings.

Motivation & Objective

• Address the gap in high-quality Spanish language models by creating large-scale pretrained models trained on a massive Spanish corpus.
• Develop and release RoBERTa-based encoder and GPT-2-based decoder models for Spanish.
• Create and evaluate a new Spanish QA dataset (SQAC) and provide extensive downstream task benchmarks.

Proposed method

• Build a large pretraining corpus from 2009–2019 BNE crawls of .es domains with rigorous cleaning and deduplication.
• Train RoBERTa-base (RoBERTa-b) and RoBERTa-large (RoBERTa-l) using masked language modeling without next-sentence prediction on a 512-token sequence length.
• Train GPT-2 (gpt2) and GPT-2 large (gpt2-large) with language modeling objectives on the same corpus.
• Fine-tune models on 9 downstream tasks with standardized HuggingFace-based procedures and grid-search hyperparameter tuning.
• Introduce SQAC, a Spanish extractive QA dataset with 18,817 questions over 6,247 contexts, to enable QA evaluation in Spanish.

Figure 1: Perplexity curves for GPT2 model.

Experimental results

Research questions

• RQ1 What is the impact of training Spanish language models on a large, diverse, non-Wikipedia corpus for downstream NLP tasks?
• RQ2 How do RoBERTa-based encoders and GPT-2-based decoders compare to existing Spanish and multilingual models on standard benchmarks?
• RQ3 Does a new Spanish QA dataset (SQAC) enable robust evaluation and development of Spanish QA systems?

Key findings

• RoBERTa-large generally achieves the best or near-best performance across most tasks, with notable gains on CAPITEL-NERC, STS, and SQAC relative to baselines.
• RoBERTa-base and RoBERTa-large outperform multilingual mBERT and several Spanish baselines (BETO, BERTIN, ELECTRA) on multiple tasks.
• The GAP between Wikipedia-trained models and non-Wikipedia data appears sizable on non-Wikipedia datasets (CAPITEL-NERC, STS, SQAC).
• UD-POS and CAPITEL-POS show very high accuracy across models, with RoBERTa variants leading or matching top baselines.
• SQAC provides a high-quality, original-Spanish QA benchmark with strong human agreement, filling a gap for Spanish extractive QA evaluation.
• GPT-2 models were evaluated with perplexity curves due to dataset limitations rather than task-based benchmarks.