[Paper Review] Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Muhammad Arslan Manzoor, Sarah Albarri+4 more|arXiv (Cornell University)|Feb 1, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This survey provides a comprehensive overview of multimodal representation learning, covering the evolution of deep learning architectures, pretraining strategies, and unified models for vision, language, and audio. It highlights transformer-based architectures, pretraining objectives, benchmark datasets, and future directions, emphasizing cross-modal alignment, knowledge infusion, and efficiency improvements for multimodal tasks.

ABSTRACT

Multimodality Representation Learning, as a technique of learning to embed information from different modalities and their correlations, has achieved remarkable success on a variety of applications, such as Visual Question Answering (VQA), Natural Language for Visual Reasoning (NLVR), and Vision Language Retrieval (VLR). Among these applications, cross-modal interaction and complementary information from different modalities are crucial for advanced models to perform any multimodal task, e.g., understand, recognize, retrieve, or generate optimally. Researchers have proposed diverse methods to address these tasks. The different variants of transformer-based architectures performed extraordinarily on multiple modalities. This survey presents the comprehensive literature on the evolution and enhancement of deep learning multimodal architectures to deal with textual, visual and audio features for diverse cross-modal and modern multimodal tasks. This study summarizes the (i) recent task-specific deep learning methodologies, (ii) the pretraining types and multimodal pretraining objectives, (iii) from state-of-the-art pretrained multimodal approaches to unifying architectures, and (iv) multimodal task categories and possible future improvements that can be devised for better multimodal learning. Moreover, we prepare a dataset section for new researchers that covers most of the benchmarks for pretraining and finetuning. Finally, major challenges, gaps, and potential research topics are explored. A constantly-updated paperlist related to our survey is maintained at https://github.com/marslanm/multimodality-representation-learning.

Motivation & Objective

• To systematically review the evolution of task-specific and transformer-based multimodal architectures for vision, language, and audio.
• To analyze pretraining types, objectives, and their role in improving generalization across downstream multimodal tasks.
• To present unified architectures that reduce fine-tuning overhead and computational complexity across diverse tasks.
• To compile a comprehensive dataset section with benchmarks for pretraining and fine-tuning to support new researchers.
• To identify key challenges, open gaps, and future research directions in multimodal representation learning, including multilingual and audio integration.

Proposed method

• Surveying recent literature on multimodal representation learning with a focus on transformer-based models and their architectural evolution.
• Classifying pretraining types (e.g., masked language modeling, contrastive learning) and analyzing their objectives in cross-modal alignment.
• Comparing state-of-the-art pretrained models (e.g., ViLBERT, CLIP, Flamingo) and unified architectures (e.g., unified vision-language-audio encoders).
• Introducing a taxonomy of deep multimodal architectures and complex multimodal applications based on modality combinations and task types.
• Proposing methodological improvements such as prompt tuning, knowledge distillation, and embedding-level masking to enhance alignment and efficiency.
• Highlighting the need for audio stream integration and multilingual pretraining to expand the scope of current models.

Figure 1. The pie chart on the left represents the percentage of papers for each section included in this survey. The pie chart in the center represents the percentage of papers for each multimodal application. The rightmost figure expresses the growth of deep learning-based multimodal papers in the

Experimental results

Research questions

• RQ1 How have transformer-based architectures evolved to unify multimodal representation learning across vision, language, and audio?
• RQ2 What are the most effective pretraining objectives and strategies for improving cross-modal alignment and generalization?
• RQ3 To what extent can unified architectures eliminate the need for task-specific fine-tuning and reduce computational costs?
• RQ4 What are the key challenges in aligning multiple modalities—especially audio—during pretraining and inference?
• RQ5 How can external knowledge, pruning, quantization, and early evaluation strategies improve the efficiency and performance of multimodal models?

Key findings

• Transformer-based architectures have significantly advanced multimodal representation learning by enabling joint processing of vision, language, and audio in a shared representation space.
• Pretraining with contrastive and masked objectives improves cross-modal alignment, with models like CLIP and ViLBERT achieving state-of-the-art performance on VLR and VQA benchmarks.
• Knowledge distillation improved ViT-BERT performance by 10% on SST-2 and 5% on IN-1K, demonstrating its value in enhancing representation quality.
• Embedding-level masking was found to be the most effective strategy for aligning representations across modalities, especially in fine-grained cross-modal understanding.
• Audio stream integration remains underexplored in current surveys and pretraining frameworks, despite its semantic richness and utility in tasks like AVSR and sentiment analysis.
• Future research should prioritize multilingual multimodal datasets and unified architectures that reduce computational cost through techniques like prompt tuning and model compression.

Figure 2. The figure exhibit the difference in the pipeline of Pretraining-Finetuning architecture and the task-specific architecture. Different modes of input in Pretraining show that multimodal data is passed to the model. The Preprocessing module formats each modal input according to the model. T