[Paper Review] On the Multilingual Capabilities of Very Large-Scale English Language Models

Jordi Armengol-Estapé, Ona De Gibert Bonet+1 more|arXiv (Cornell University)|Aug 30, 2021

Topic Modeling4 citations

TL;DR

This paper investigates the zero-shot multilingual capabilities of GPT-3, a large-scale English-only language model, focusing on Catalan—a low-resource language comprising only 0.01798% of the training data. Despite minimal exposure, GPT-3 demonstrates strong performance in both natural language generation and extractive question-answering, with F1 scores reaching 65.8% on the largest model variant, indicating significant cross-lingual transfer even in low-resource settings.

ABSTRACT

Generative Pre-trained Transformers (GPTs) have recently been scaled to unprecedented sizes in the history of machine learning. These models, solely trained on the language modeling objective, have been shown to exhibit outstanding few-shot learning capabilities in a number of different tasks. Nevertheless, aside from anecdotal experiences, little is known regarding their multilingual capabilities, given the fact that the pre-training corpus is almost entirely composed of English text. In this work, we investigate the multilingual skills of GPT-3, focusing on one language that barely appears in the pre-training corpus, Catalan, which makes the results especially meaningful; we assume that our results may be relevant for other languages as well. We find that the model shows an outstanding performance, particularly in generative tasks, with predictable limitations mostly in language understanding tasks but still with remarkable results given the zero-shot scenario. We investigate its potential and limits in extractive question-answering and natural language generation, as well as the effect of scale in terms of model size.

Motivation & Objective

• To evaluate the zero-shot multilingual capabilities of GPT-3 in a low-resource language like Catalan, which constitutes only 0.01798% of the pre-training corpus.
• To assess whether very large-scale English language models can generalize to other languages without fine-tuning or explicit multilingual pre-training.
• To investigate the impact of model scale on cross-lingual transfer performance in natural language understanding and generation tasks.
• To explore the practical usability of GPT-3 for non-English languages, particularly in terms of computational cost and tokenization inefficiency.

Proposed method

• The study evaluates GPT-3 across multiple model sizes (text-davinci-002, text-curie-001, etc.) using zero-shot prompting for both question-answering and text generation tasks in Catalan.
• For extractive question-answering, the model is prompted with context and question pairs, and predictions are evaluated using F1 and exact match scores.
• For natural language generation, GPT-3 is prompted with headlines or partial sentences to generate coherent Catalan text, which is then evaluated via human annotation on a 5-point scale.
• Human evaluation involves three annotators per sample to ensure reliability, with statistical significance tested using a t-test.
• The analysis compares GPT-3’s performance to a human-written control set, focusing on fluency, coherence, and semantic accuracy.
• Scaling curves are plotted to analyze the relationship between model size and performance in zero-shot cross-lingual transfer.

Experimental results

Research questions

• RQ1 Can GPT-3 perform zero-shot natural language understanding and generation in Catalan despite its minimal presence in the pre-training data?
• RQ2 How does model size affect the performance of GPT-3 in zero-shot multilingual tasks for low-resource languages?
• RQ3 How does GPT-3’s performance in Catalan compare to human-generated text in terms of fluency and coherence?
• RQ4 What are the main failure modes and error patterns in GPT-3’s Catalan outputs?
• RQ5 To what extent can linguistic proximity and structural universals explain the observed cross-lingual transfer in GPT-3?

Key findings

• GPT-3 achieves an F1 score of 65.8% on extractive question-answering in Catalan using the largest model variant (text-davinci-002), indicating strong zero-shot NLU capabilities.
• In natural language generation, GPT-3’s outputs received a mean human evaluation score of 3.83 out of 5, significantly lower than the human control’s 4.49 (p = 0.00026), but still high given the zero-shot setting.
• Despite lower average scores, 33.33% of GPT-3’s generated sentences were rated above the human control average, and 21.6% were syntactically correct but semantically idiosyncratic.
• The model exhibits remarkable consistency in generating text in the Valencian variant of Catalan when prompted with region-specific headlines, suggesting implicit adaptation to dialectal variation.
• Scaling curves show a steep increase in performance with model size, even when pre-training data for Catalan remains constant, indicating strong scaling laws in cross-lingual transfer.
• Smaller GPT-3 variants are computationally impractical for Catalan due to inefficient tokenization and long sequence lengths, highlighting a key bottleneck for low-resource languages.