[论文解读] Foundation Models for Natural Language Processing -- Pre-trained Language Models Integrating Media

Gerhard Paaß, Sven Giesselbach|arXiv (Cornell University)|Feb 16, 2023

Topic Modeling被引用 10

一句话总结

综述性研究：在自然语言处理（NLP）中引入 Foundation Models，详细介绍预训练语言模型、它们的体系结构、训练策略以及跨多模态能力，并讨论评估、伦理和社会影响。

ABSTRACT

This open access book provides a comprehensive overview of the state of the art in research and applications of Foundation Models and is intended for readers familiar with basic Natural Language Processing (NLP) concepts. Over the recent years, a revolutionary new paradigm has been developed for training models for NLP. These models are first pre-trained on large collections of text documents to acquire general syntactic knowledge and semantic information. Then, they are fine-tuned for specific tasks, which they can often solve with superhuman accuracy. When the models are large enough, they can be instructed by prompts to solve new tasks without any fine-tuning. Moreover, they can be applied to a wide range of different media and problem domains, ranging from image and video processing to robot control learning. Because they provide a blueprint for solving many tasks in artificial intelligence, they have been called Foundation Models. After a brief introduction to basic NLP models the main pre-trained language models BERT, GPT and sequence-to-sequence transformer are described, as well as the concepts of self-attention and context-sensitive embedding. Then, different approaches to improving these models are discussed, such as expanding the pre-training criteria, increasing the length of input texts, or including extra knowledge. An overview of the best-performing models for about twenty application areas is then presented, e.g., question answering, translation, story generation, dialog systems, generating images from text, etc. For each application area, the strengths and weaknesses of current models are discussed, and an outlook on further developments is given. In addition, links are provided to freely available program code. A concluding chapter summarizes the economic opportunities, mitigation of risks, and potential developments of AI.

研究动机与目标

• 解释基于注意力的语言模型的核心体系结构类型（自编码器、自回归、编码器-解码器）及它们如何形成预训练语言模型（PLMs）。
• 描述使大型 PLMs 能跨语言和模态执行多种任务的预训练、微调和提示范式。
• 总结 Foundation Models 如何将 PLMs 扩展到多媒体输入（文本、语音、图像、视频）及其潜在的社会与经济影响。
• 评估与 Foundation Models 相关的方法论考量，包括知识整合、基准测试、可重复性，以及伦理关注点。

提出的方法

• 对体系结构类型进行分类（基于自编码器、.autoregressive、序列到序列）并描述它们的预训练目标。
• 讨论缩放定律：通过增加参数、数据和计算资源来提升性能。
• 解释多语言训练以及引入额外知识以提升预测能力。
• 描述任务适应与小样本学习的提示与微调策略。
• 概述跨媒体标记化（文本、图像块、视频片段）以及跨模态自注意力的应用。
• 讨论 Foundation Models 的评估、可靠性与可重复性。

实验结果

研究问题

• RQ1 Foundation Models 在体系结构、训练和能力方面与传统 PLMs 有何不同？
• RQ2 哪些策略（预训练任务、数据规模、多语言性、知识注入）推动 Foundation Models 的性能提升？
• RQ3 如何将 Foundation Models 扩展到多模态数据和跨媒体任务？
• RQ4 哪些基准测试与探测方法揭示 Foundation Models 的知识、推理和局限性，以及这些结果的可重复性如何？

主要发现

• Foundation Models 在增加参数、数据和计算资源的条件下达到高性能。
• 跨媒体表示使在统一模型中处理文本、语音、图像和视频成为可能。
• 大型 PLMs 可以通过提示和小样本学习进行指令化，借助最少的微调实现任务适应。
• 基准测试和探测揭示知识与推理能力，但也提出关于可靠性与可重复性的问题。
• 存在潜在危害，包括偏见、错误信息、经济垄断和失业，促使监管方面的考虑。