[Paper Review] Multimodal Machine Learning: A Survey and Taxonomy

Tadas Baltrušaitis, Chaitanya Ahuja+1 more|arXiv (Cornell University)|May 26, 2017

Advanced Chemical Sensor TechnologiesEngineering105 citations

TL;DR

This paper proposes a comprehensive taxonomy for multimodal machine learning, moving beyond traditional early and late fusion to identify five core challenges: representation, translation, alignment, fusion, and co-learning. It synthesizes recent advances across these areas, offering a structured framework to guide future research and improve understanding of multimodal AI systems.

ABSTRACT

Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.

Motivation & Objective

• To address the growing complexity and heterogeneity of multimodal data in AI by identifying core technical challenges beyond traditional fusion methods.
• To provide a unified taxonomy that structures recent advances in multimodal machine learning for better research navigation and progress tracking.
• To highlight the interplay between modalities—language, vision, and audio—and their joint learning for improved AI understanding.
• To identify unresolved problems in multimodal learning, particularly in representation learning, cross-modal alignment, and zero-shot transfer.
• To support future research by organizing existing work under five key challenges: representation, translation, alignment, fusion, and co-learning.

Proposed method

• Proposes a five-tier taxonomy of multimodal machine learning challenges: representation, translation, alignment, fusion, and co-learning.
• Analyzes recent methods under each challenge category, including deep learning architectures, attention mechanisms, and contrastive learning for cross-modal alignment.
• Reviews techniques such as late fusion (early fusion), multimodal autoencoders, and pivot-based models (e.g., Bridge Correlational Neural Network) for non-parallel data.
• Introduces conceptual grounding as a method to learn shared latent spaces between modalities using visual or acoustic signals.
• Applies zero-shot learning (ZSL) via semantic space mapping, where unseen visual classes are predicted using textual embeddings from a pre-trained semantic space.
• Uses large-scale datasets like MS-COCO, TRECVID, and IEMOCAP to evaluate and illustrate the taxonomy across applications.

Experimental results

Research questions

• RQ1 How can multimodal data from heterogeneous sources (e.g., text, images, audio) be effectively represented in a shared, meaningful space?
• RQ2 What methods enable accurate mapping between modalities when one-to-one correspondence is ambiguous or absent?
• RQ3 How can temporal and semantic alignments be established between elements across modalities, such as video frames and spoken words?
• RQ4 What are the most effective strategies for fusing multimodal signals to improve prediction performance in tasks like speech recognition or action classification?
• RQ5 In what ways can knowledge be transferred across modalities to improve learning, especially when one modality has limited labeled data?

Key findings

• Representation learning in multimodal systems is critical for capturing complementary and redundant information across modalities, with deep neural networks enabling effective joint embeddings.
• Translation between modalities is inherently ambiguous due to open-ended mappings, but neural network-based models with attention mechanisms can learn plausible cross-modal mappings.
• Alignment of sub-elements across modalities—such as aligning image regions with textual phrases—can be achieved using contrastive learning and attention-based mechanisms, improving performance in visual question answering and captioning.
• Fusion strategies that dynamically weigh modality contributions based on confidence and noise levels lead to improved robustness, especially in low-signal environments.
• Co-learning techniques such as zero-shot learning and co-training enable models to generalize to unseen classes by leveraging semantic or visual attributes from related modalities, with Socher et al. achieving 75% accuracy on unseen visual classes using semantic space transfer.
• Conceptual grounding using visual or auditory signals significantly improves performance on tasks involving metaphor detection, semantic similarity, and language understanding, particularly when grounding is relevant to the task domain.