[Paper Review] BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Krahmer, Emiel, Clouth, Felix+6 more|arXiv (Cornell University)|Nov 9, 2022

Topic Modeling278 citations

TL;DR

BLOOM is a 176-billion-parameter, open-access, multilingual language model developed by the BigScience Workshop to enable research and application across diverse languages and tasks. Trained on a curated, multilingual dataset using a mixture-of-experts architecture, BLOOM achieves strong zero-shot and few-shot performance across 46 languages and 255 tasks, demonstrating state-of-the-art results in several benchmarks while maintaining transparency through open weights and detailed documentation.

ABSTRACT

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

Motivation & Objective

• To develop a large-scale, open-access multilingual language model that supports diverse research and application needs across multiple languages.
• To enable zero-shot and few-shot generalization across a broad range of natural language tasks without fine-tuning.
• To ensure transparency and reproducibility by releasing the full model weights, training data, and evaluation protocols.
• To mitigate risks associated with proprietary LLMs by creating a community-driven, ethically guided alternative with documented broader impacts.
• To support multilingual research by training on a curated dataset spanning 46 languages, including low-resource languages.

Proposed method

• Training BLOOM on a diverse, curated dataset of 46 languages using a mixture-of-experts (MoE) transformer architecture with 176 billion parameters.
• Employing a combination of instruction tuning and few-shot prompting to enable zero-shot and few-shot generalization across 255 tasks.
• Using a subword tokenizer trained on the full multilingual dataset to ensure robust tokenization across languages.
• Leveraging the Jean Zay supercomputer at IDRIS for training, with infrastructure and compute support from Hugging Face, CoreWeave, and EleutherAI.
• Applying systematic prompt engineering and evaluation protocols to ensure consistency and reliability across tasks and languages.
• Conducting comprehensive evaluation across multiple benchmarks, including Big-Bench, MMLU, and GLUE, to measure zero-shot and few-shot performance.

Experimental results

Research questions

• RQ1 Can a large, open-access multilingual language model generalize effectively across diverse languages and tasks without fine-tuning?
• RQ2 How does BLOOM's performance compare to proprietary models like GPT-3 and PaLM across zero-shot and few-shot benchmarks?
• RQ3 To what extent can BLOOM support low-resource languages and underrepresented linguistic varieties?
• RQ4 What are the broader societal, ethical, and environmental impacts of training and releasing such a large model openly?
• RQ5 How does the model's architecture and training objective influence its zero-shot generalization capabilities?

Key findings

• BLOOM achieves state-of-the-art zero-shot performance on several benchmarks, including Big-Bench and MMLU, outperforming smaller models and matching or exceeding some proprietary models in specific tasks.
• The model demonstrates strong few-shot generalization across 255 diverse natural language tasks spanning 46 languages, including low-resource and morphologically complex languages.
• BLOOM's performance is competitive with proprietary models like GPT-3 and PaLM in zero-shot settings, particularly in reasoning and commonsense tasks.
• The model's open weights and training data enable reproducibility and community-driven development, supporting transparency and ethical AI research.
• The training process was carbon-efficient relative to model size, with a documented carbon footprint and mitigation strategies implemented throughout the project lifecycle.
• Evaluation across diverse linguistic and cultural contexts revealed robust performance across both high- and low-resource languages, with notable capabilities in multilingual reasoning and code generation.