[论文解读] PaLM 2 Technical Report

Rohan Anil, Andrew M. Dai+126 more|arXiv (Cornell University)|May 17, 2023

Topic Modeling被引用 159

一句话总结

PaLM 2 是一种基于 Transformer 的语言模型，在多语言、推理和效率方面相较 PaLM 有改进，具备混合预训练目标、多语言数据以及推理时对毒性进行控制。

ABSTRACT

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.

研究动机与目标

• 在英语及多语言任务上展示 PaLM 2 的前沿语言建模能力。
• 评估缩放定律以及数据混合对性能与效率的影响。
• 评估多语言能力、推理、编码、翻译和生成在多样数据集上的表现。
• 审查负责任AI方面的内容，包括记忆、毒性控制和偏见。
• 描述设计考虑、训练数据和面向下游研究者的评估方法。

提出的方法

• 使用基于 Transformer 的架构，结合受 UL2 启发的经过调整的预训练目标混合。
• 在多样化的多语言预训练语料上进行训练，包括网络、书籍、代码、数学，以及并行多语言数据。
• 应用计算最优缩放分析，以在 FLOPs 预算内估计最佳模型规模和数据需求。
• 在语言能力考试、分类/问答、推理、编码、翻译和生成基准测试中进行评估。
• 引入推理时控制标记，以在不牺牲其他能力的情况下管理毒性。
• 在附录和 D 部分提供对记忆、偏见和负责任AI考虑的详细分析。

Figure 1: Performance of PaLM 2 and PaLM on the latest available professional language proficiency exams. We used exams for each language to test a C2 (mastery or advanced professional) level proficiency following the CEFR definition. We used the transcripts of the listening portion of the exams and

实验结果

研究问题

• RQ1 在固定算力下，数据量和模型规模如何随着计算资源的分配而扩展以优化 PaLM 2 的性能？
• RQ2 更丰富的多语言预训练混合是否在不损害英语能力的前提下提升多语言与英语的性能？
• RQ3 PaLM 2 在跨语言的推理、编码、翻译与生成方面有哪些能力，并与先前的模型如 PaLM 和 GPT-4 相比？
• RQ4 推理时毒性控制及其他负责任AI措施在真实世界任务中的有效性如何？
• RQ5 PaLM 2 的预训练数据在记忆、偏见和数据集范围方面的含义是什么？

主要发现

• PaLM 2 在语言、推理、翻译和编码任务上超越 PaLM，且适用于不同模型规模。
• 缩放定律指出，在固定计算资源下，训练数据和模型规模应大致按等比例增长以实现最佳性能。
• 更大版本的 PaLM 2 在多语言 TyDi QA 和零SHOT/无上下文设置上取得强劲结果，尤其是对资源较少的语言。
• 推理时控制标记可以在对其他能力影响极小的情况下调控毒性。
• PaLM 2 在专业语言考试和 BIG-Bench Hard 任务上表现强劲，在多语言和推理基准测试中获得显著提升。

Figure 3: Describing the meaning of a transliterated Persian proverb and its equivalent in several other languages. The Persian proverb is an equivalent of ”No Pain, No Gain” in English. The model is capable of going beyond literal translation and mentions proverbs that are used in the target langua