[论文解读] Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Muhammad Arslan Manzoor, Sarah Albarri+4 more|arXiv (Cornell University)|Feb 1, 2023

Multimodal Machine Learning Applications被引用 4

一句话总结

本综述全面概述了多模态表征学习的发展，涵盖深度学习架构的演进、预训练策略以及视觉、语言和语音的统一模型。文章重点介绍了基于Transformer的架构、预训练目标、基准数据集及未来方向，强调了跨模态对齐、知识注入和效率提升在多模态任务中的重要性。

ABSTRACT

Multimodality Representation Learning, as a technique of learning to embed information from different modalities and their correlations, has achieved remarkable success on a variety of applications, such as Visual Question Answering (VQA), Natural Language for Visual Reasoning (NLVR), and Vision Language Retrieval (VLR). Among these applications, cross-modal interaction and complementary information from different modalities are crucial for advanced models to perform any multimodal task, e.g., understand, recognize, retrieve, or generate optimally. Researchers have proposed diverse methods to address these tasks. The different variants of transformer-based architectures performed extraordinarily on multiple modalities. This survey presents the comprehensive literature on the evolution and enhancement of deep learning multimodal architectures to deal with textual, visual and audio features for diverse cross-modal and modern multimodal tasks. This study summarizes the (i) recent task-specific deep learning methodologies, (ii) the pretraining types and multimodal pretraining objectives, (iii) from state-of-the-art pretrained multimodal approaches to unifying architectures, and (iv) multimodal task categories and possible future improvements that can be devised for better multimodal learning. Moreover, we prepare a dataset section for new researchers that covers most of the benchmarks for pretraining and finetuning. Finally, major challenges, gaps, and potential research topics are explored. A constantly-updated paperlist related to our survey is maintained at https://github.com/marslanm/multimodality-representation-learning.

研究动机与目标

• 系统性回顾视觉、语言和语音领域中任务特定及基于Transformer的多模态架构的演进。
• 分析预训练类型、目标及其在提升下游多模态任务泛化能力中的作用。
• 提出统一架构，以减少不同任务中微调的开销和计算复杂度。
• 整理全面的数据集部分，包含用于预训练和微调的基准，以支持新研究者。
• 识别多模态表征学习中的关键挑战、开放性问题及未来研究方向，包括多语言和语音的整合。

提出的方法

• 聚焦基于Transformer的模型及其架构演进，系统调研近期多模态表征学习相关文献。
• 对预训练类型（如掩码语言建模、对比学习）进行分类，并分析其在跨模态对齐中的目标。
• 对比最先进预训练模型（如ViLBERT、CLIP、Flamingo）和统一架构（如统一视觉-语言-语音编码器）。
• 基于模态组合和任务类型，提出深度多模态架构及复杂多模态应用的分类体系。
• 提出方法学改进，如提示微调、知识蒸馏和嵌入级掩码，以增强对齐效果和效率。
• 强调需要整合语音流和多语言预训练，以拓展当前模型的应用范围。

Figure 1. The pie chart on the left represents the percentage of papers for each section included in this survey. The pie chart in the center represents the percentage of papers for each multimodal application. The rightmost figure expresses the growth of deep learning-based multimodal papers in the

实验结果

研究问题

• RQ1 基于Transformer的架构如何演进以统一视觉、语言和语音的多模态表征学习？
• RQ2 哪些预训练目标和策略在提升跨模态对齐和泛化能力方面最为有效？
• RQ3 统一架构在多大程度上可消除对任务特定微调的需求并降低计算成本？
• RQ4 在预训练和推理过程中，对齐多种模态（尤其是语音）面临哪些关键挑战？
• RQ5 外部知识注入、剪枝、量化和早期评估策略如何提升多模态模型的效率和性能？

主要发现

• 基于Transformer的架构通过在共享表征空间中联合处理视觉、语言和语音，显著推动了多模态表征学习的发展。
• 采用对比学习和掩码目标进行预训练可提升跨模态对齐效果，CLIP和ViLBERT等模型在VLR和VQA基准上达到最先进性能。
• 知识蒸馏使ViT-BERT在SST-2上性能提升10%，在IN-1K上提升5%，证明其在提升表征质量方面的价值。
• 嵌入级掩码被证明是提升模态间表征对齐效果的最有效策略，尤其在细粒度跨模态理解任务中表现突出。
• 尽管语音在AVSR和情感分析等任务中具有丰富的语义信息，但当前综述和预训练框架中对语音流的整合仍研究不足。
• 未来研究应优先关注多语言多模态数据集，以及通过提示微调和模型压缩等技术降低计算成本的统一架构。

Figure 2. The figure exhibit the difference in the pipeline of Pretraining-Finetuning architecture and the task-specific architecture. Different modes of input in Pretraining show that multimodal data is passed to the model. The Preprocessing module formats each modal input according to the model. T