[Paper Review] PaLM 2 Technical Report

Rohan Anil, Andrew M. Dai+126 more|arXiv (Cornell University)|May 17, 2023

Topic Modeling159 citations

TL;DR

PaLM 2 is a Transformer-based language model with multilingual, reasoning, and efficiency improvements over PaLM, featuring a mixture of pre-training objectives, multilingual data, and inference-time control over toxicity.

ABSTRACT

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.

Motivation & Objective

• Demonstrate state-of-the-art language modeling with PaLM 2 across English and multilingual tasks.
• Evaluate scaling laws and the impact of data mix on performance and efficiency.
• Assess multilingual capabilities, reasoning, coding, translation, and generation across diverse datasets.
• Examine responsible AI aspects including memory, toxicity control, and bias.
• Describe design considerations, training data, and evaluation methodology for downstream researchers.

Proposed method

• Use a Transformer-based architecture with a tuned mixture of pre-training objectives inspired by UL2.
• Train on a diverse, multilingual pre-training corpus including web, books, code, mathematics, and parallel multilingual data.
• Apply compute-optimal scaling analysis to estimate optimal model size and data requirements across FLOPs budgets.
• Evaluate on language proficiency exams, classification/QA, reasoning, coding, translation, and generation benchmarks.
• Incorporate inference-time control tokens to manage toxicity without sacrificing other capabilities.
• Provide detailed analysis of memorization, bias, and responsible AI considerations in Appendix and D sections.

Figure 1: Performance of PaLM 2 and PaLM on the latest available professional language proficiency exams. We used exams for each language to test a C2 (mastery or advanced professional) level proficiency following the CEFR definition. We used the transcripts of the listening portion of the exams and

Experimental results

Research questions

• RQ1 How do data size and model size scale with fixed compute to optimize PaLM 2 performance?
• RQ2 Does a more multilingual pre-training mixture improve multilingual and English performance without harming English capabilities?
• RQ3 What are PaLM 2’s capabilities in reasoning, coding, translation, and generation across languages, and how do they compare to prior models like PaLM and GPT-4?
• RQ4 How effective are inference-time toxicity controls and other responsible AI measures in real-world tasks?
• RQ5 What are the memorization, bias, and dataset-spectrum implications of PaLM 2’s pre-training data?

Key findings

• PaLM 2 outperforms PaLM across language, reasoning, translation, and coding tasks across model sizes.
• Scaling laws indicate that training data and model size should grow in roughly equal proportions for optimal performance under fixed compute.
• A larger PaLM 2 variant achieves strong results on multilingual TyDi QA and zero-shot/no-context settings, particularly for lower-resource languages.
• Inference-time control tokens can modulate toxicity with minimal impact on other capabilities.
• PaLM 2 shows robust performance on professional language exams and BIG-Bench Hard tasks, with notable gains in multilingual and reasoning benchmarks.

Figure 3: Describing the meaning of a transliterated Persian proverb and its equivalent in several other languages. The Persian proverb is an equivalent of ”No Pain, No Gain” in English. The model is capable of going beyond literal translation and mentions proverbs that are used in the target langua