[Paper Review] The Evolution of Multimodal Model Architectures

Shakti N. Wadekar, Abhishek Chaurasia+2 more|arXiv (Cornell University)|May 28, 2024

Speech and dialogue systems4 citations

TL;DR

This paper introduces a novel taxonomy of four distinct multimodal model architectures—Type-A (standard cross-attention), Type-B (custom fusion layers), Type-C (non-tokenized early fusion), and Type-D (tokenized early fusion)—to systematize the evolving multimodal landscape. It identifies Type-C and Type-D as dominant for any-to-any multimodal generation, offering a structured framework for model selection based on trade-offs in complexity, scalability, and training objectives.

ABSTRACT

This work uniquely identifies and characterizes four prevalent multimodal model architectural patterns in the contemporary multimodal landscape. Systematically categorizing models by architecture type facilitates monitoring of developments in the multimodal domain. Distinct from recent survey papers that present general information on multimodal architectures, this research conducts a comprehensive exploration of architectural details and identifies four specific architectural types. The types are distinguished by their respective methodologies for integrating multimodal inputs into the deep neural network model. The first two types (Type A and B) deeply fuses multimodal inputs within the internal layers of the model, whereas the following two types (Type C and D) facilitate early fusion at the input stage. Type-A employs standard cross-attention, whereas Type-B utilizes custom-designed layers for modality fusion within the internal layers. On the other hand, Type-C utilizes modality-specific encoders, while Type-D leverages tokenizers to process the modalities at the model's input stage. The identified architecture types aid the monitoring of any-to-any multimodal model development. Notably, Type-C and Type-D are currently favored in the construction of any-to-any multimodal models. Type-C, distinguished by its non-tokenizing multimodal model architecture, is emerging as a viable alternative to Type-D, which utilizes input-tokenizing techniques. To assist in model selection, this work highlights the advantages and disadvantages of each architecture type based on data and compute requirements, architecture complexity, scalability, simplification of adding modalities, training objectives, and any-to-any multimodal generation capability.

Motivation & Objective

• To address the lack of systematic categorization in existing surveys by identifying and formalizing four distinct architectural patterns in multimodal models.
• To enable better monitoring and tracking of advancements in multimodal AI by providing a standardized taxonomy based on fusion strategy and integration stage.
• To support the design and selection of any-to-any multimodal models by comparing architectural trade-offs in data efficiency, scalability, and training complexity.
• To highlight the rising dominance of Type-C and Type-D architectures in next-generation multimodal systems, especially for end-to-end and agent-augmented generation.
• To provide a comprehensive, model-agnostic framework for researchers and practitioners to evaluate and select architectures based on specific application constraints.

Proposed method

• Proposes a four-type taxonomy: Type-A (standard cross-attention), Type-B (custom fusion layers), Type-C (non-tokenized early fusion), and Type-D (tokenized early fusion) based on fusion timing and mechanism.
• Categorizes state-of-the-art models (e.g., Flamingo, CLIP, Unified-IO, CoDI, NExT-GPT, ModaVerse) into the four architectural types using architectural and fusion strategy analysis.
• Analyzes architectural components such as modality-specific encoders (Type-C), tokenizers (Type-D), and attention mechanisms (Type-A/B) to define structural distinctions.
• Evaluates each architecture type across key dimensions: data and compute requirements, complexity, scalability, modality extensibility, training objectives, and any-to-any generation capability.
• Introduces a visual taxonomy (Figure 2) and timeline (Figure 1) to map model evolution and architectural trends across the multimodal space.
• Explores emerging alternatives like state space models (SSMs), such as VL-Mamba and Cobra, which align structurally with Type-C and may enable future SSM-based any-to-any models.

Experimental results

Research questions

• RQ1 What are the four distinct architectural patterns underlying contemporary multimodal models, and how do they differ in fusion strategy and integration timing?
• RQ2 Why are Type-C and Type-D architectures currently favored for building any-to-any multimodal models, and what advantages do they offer over deep fusion approaches?
• RQ3 How do architectural choices impact training complexity, scalability, and the ability to support diverse modalities in multimodal generation?
• RQ4 What are the performance trade-offs between end-to-end trainable models (Type-C, Type-D) and hybrid agent-based approaches in any-to-any multimodal systems?
• RQ5 Can emerging architectures like state space models (SSMs) be integrated into existing taxonomy types, and what potential do they hold for future multimodal model design?

Key findings

• Type-D models such as Unified-IO-2 achieve a FID of 13.39 on image generation and a CIDEr score of 48.8 on video captioning, demonstrating strong performance in end-to-end multimodal generation.
• Type-C models like CoDI achieve a lower FID of 11.26 and a higher CIDEr of 78.9 on video captioning, indicating superior image generation quality and better alignment with reference captions compared to Type-D.
• NExT-GPT, a Type-C model, achieves a CIDEr score of 156.7 on audio generation and 0.802 on CLIPSIM, outperforming other models in audio-to-video and cross-modal alignment tasks.
• ModaVerse, which combines Type-C with agents, achieves a FID of 11.24 and a CIDEr of 151.4, showing that non-end-to-end approaches can achieve competitive performance despite lacking full end-to-end training.
• The study identifies that Type-C and Type-D are the two dominant architectural paradigms for any-to-any multimodal models, with Type-C gaining traction as a non-tokenizing alternative to Type-D.
• State space models (SSMs) such as VL-Mamba and Cobra show architectural similarity to Type-C, suggesting potential for future SSM-based any-to-any models, though no such models have been developed yet.