[Paper Review] mT5: A massively multilingual pre-trained text-to-text transformer

Linting Xue, Noah Constant+6 more|arXiv (Cornell University)|Oct 22, 2020

Topic ModelingComputer Science49 references601 citations

TL;DR

mT5 extends T5 to 101 languages with a multilingual pre-training dataset (mC4) and shows state-of-the-art results on multilingual benchmarks, addressing accidental translation in zero-shot generation with a domain-preserving fine-tuning technique.

ABSTRACT

The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

Motivation & Objective

• Develop a massively multilingual variant of T5 with minimal deviation from the original recipe.
• Create a multilingual pre-training dataset (mC4) covering 101 languages from Common Crawl data.
• Evaluate mT5 on xtreme multilingual benchmarks for classification, QA, and NER.
• Investigate and mitigate accidental translation in zero-shot generation through domain-preserving fine-tuning.
• Provide open-source code and pre-trained checkpoints to the community.

Proposed method

• Extend the T5.1.1 recipe (GeGLU activations, larger d_model and d_ff, no dropout during pre-training).
• Build mC4 by language-detecting and filtering Common Crawl data to include 101 languages with a 70% confidence threshold and 10,000+ pages per language.
• Sample multilingual data during pre-training with p(L) ∝ |L|^α to boost low-resource languages (α tuned to 0.3).
• Use a 250k SentencePiece vocabulary with language-specific coverage and byte-fallback for large scripts.
• Fine-tune and evaluate across xtreme tasks by casting all tasks to a text-to-text format; explore zero-shot, translate-train, and in-language multitask settings.
• Address zero-shot accidental translation by domain-adaptive pre-training (mixing in mC4 during fine-tuning) and reducing language sampling bias (α lowered to 0.1).

Experimental results

Research questions

• RQ1 Can a T5-style multilingual model trained on a broad, multilingual corpus achieve state-of-the-art results across many languages?
• RQ2 How does model capacity affect cross-lingual transfer and the need for translated or in-language data?
• RQ3 What mechanisms cause accidental translation in zero-shot generation, and can multilingual pre-training data mixed into fine-tuning mitigate it?
• RQ4 How does language sampling during pre-training affect performance on high- vs. low-resource languages?

Key findings

• mT5-XXL achieves state-of-the-art results on classification and QA benchmarks and approaches SOTA on NER.
• In xtreme tasks, translate-train with mT5 surpasses prior models; zero-shot performance approaches translate-train as model size grows.
• Larger mT5 models close the gap with English-only T5 on SQuAD, indicating capacity can mitigate cross-lingual interference.
• A small amount of multilingual pre-training data mixed into fine-tuning markedly reduces illegal/accidental translation predictions in zero-shot XQuAD for smaller models.
• Line-length filtering during data collection significantly improves XNLI accuracy (+2 points in ablations).
• Increasing α to boost high-resource languages helps some languages but harms many low-resource ones; reducing α to near-uniform reduces this bias.