[Paper Review] Aya 23: Open Weight Releases to Further Multilingual Progress

Viraat Aryabumi, John Dang+19 more|arXiv (Cornell University)|May 23, 2024

Artificial Intelligence in Healthcare and Education8 citations

TL;DR

Aya 23 releases open weights for 8B and 35B multilingual models covering 23 languages, balancing depth and breadth to improve performance over Aya 101 and competing models across discriminative, generative, and multilingual tasks.

ABSTRACT

This technical report introduces Aya 23, a family of multilingual language models. Aya 23 builds on the recent release of the Aya model (Üstün et al., 2024), focusing on pairing a highly performant pre-trained model with the recently released Aya collection (Singh et al., 2024). The result is a powerful multilingual large language model serving 23 languages, expanding state-of-art language modeling capabilities to approximately half of the world's population. The Aya model covered 101 languages whereas Aya 23 is an experiment in depth vs breadth, exploring the impact of allocating more capacity to fewer languages that are included during pre-training. Aya 23 outperforms both previous massively multilingual models like Aya 101 for the languages it covers, as well as widely used models like Gemma, Mistral and Mixtral on an extensive range of discriminative and generative tasks. We release the open weights for both the 8B and 35B models as part of our continued commitment for expanding access to multilingual progress.

Motivation & Objective

• Address data scarcity and multilinguality in LLMs by expanding language coverage beyond English.
• Investigate the impact of allocating more capacity to a smaller set of languages during pre-training.
• Evaluate multilingual and cross-lingual performance across discriminative, generative, and mathematical reasoning tasks.

Proposed method

• Base architecture: decoder-only Transformer under Cohere Command lineage with parallel attention/FFN, SwiGLU, no biases, RoPE positional embeddings, 256k BPE tokenizer.
• Instruction fine-tuning using a multilingual mix of data sources including multilingual templates, human annotations, translated data, and synthetic data to create a 1.63M example multilingual fine-tuning set.
• Training setup: 13,200 update steps, 8192 context length, Adam with cosine LR schedule, peak LR 6e-4, end LR 6e-5, batch size 64, TPUv4 with up to 128 pod slices.
• Model variants: Aya-23-8B and Aya-23-35B, with Grouped Query Attention (GQA) for the 8B model and standard attention for larger models.
• Evaluation framework: multilingual evaluation following Üstün et al. (2024) and eval-harness for discriminative tasks, multilingual MMLU, MGSM, translation, and summarization, plus safety/bias assessments.

Experimental results

Research questions

• RQ1 Can concentrating model capacity on 23 pre-training languages improve per-language performance relative to broader multilingual models?
• RQ2 How do Aya 23 models perform on discriminative, generative, translation, and mathematical reasoning tasks across 23 languages?
• RQ3 What are the safety and toxicity profiles of Aya 23 models in multilingual settings compared to a strong baseline?
• RQ4 Does open-weight release of Aya 23 meaningfully expand accessibility and research progress in multilingual NLP?

Key findings

• Aya 23-35B achieves the highest results across evaluated tasks and languages, outperforming Aya 101 and competing models.
• Aya 23-8B shows best-in-class multilingual performance among smaller models, with up to 14% improvement on discriminative tasks, up to 20% on generative tasks, and up to 41.6% in multilingual MMLU over Aya 101.
• Aya 23 models exhibit strong multilingual mathematical reasoning, with a 6.6x increase in MGSM performance over Aya 101.
• In open-ended evaluation, Aya 23 models are consistently preferred over baselines in GPT-4 judge and human assessments, especially for non-European languages.
• Safety analysis shows reduced harmful responses for Aya 23 versus Aya-101-13B, with larger models further reducing harmfulness in several languages.