[Paper Review] Recent Advances and Trends in Multimodal Deep Learning: A Review

Summaira Jabeen, Xi Li+3 more|arXiv (Cornell University)|May 24, 2021

Multimodal Machine Learning Applications172 references18 citations

TL;DR

This survey presents a comprehensive review of recent advances in multimodal deep learning (MMDL), proposing a fine-grained taxonomy of applications across image, video, text, audio, gestures, facial expressions, and physiological signals. It analyzes architectures, datasets, evaluation metrics, and open challenges, offering detailed research directions for future work in multimodal representation learning and integration.

ABSTRACT

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of past and current baseline approaches and an in-depth study of recent advancements in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning applications is proposed, elaborating on different applications in more depth. Architectures and datasets used in these applications are also discussed, along with their evaluation metrics. Last, main issues are highlighted separately for each domain along with their possible future research directions.

Motivation & Objective

• To provide a state-of-the-art review of multimodal deep learning (MMDL) across diverse modalities including image, video, text, audio, body gestures, facial expressions, and physiological signals.
• To address the limitations of unimodal learning by emphasizing the need for models that integrate multiple sensory inputs for richer, more human-like understanding.
• To propose a novel, fine-grained taxonomy of MMDL applications to better organize and classify emerging research trends.
• To analyze key architectures, benchmark datasets, and evaluation metrics used in current MMDL applications.
• To identify open research problems and suggest concrete future research directions for each application domain.

Proposed method

• Systematically categorize MMDL applications into distinct domains based on modality types and application tasks.
• Review and compare baseline and recent state-of-the-art models, including attention mechanisms, multimodal transformers, and late/fusion architectures.
• Analyze widely used datasets such as MS-COCO, IEMOCAP, MSR-VTT, and DAQUAR, highlighting their characteristics and suitability for specific tasks.
• Examine evaluation metrics like mAP, AUC, mean absolute error (MAE), and MOS to assess model performance across different MMDL tasks.
• Employ a structured analysis of neural network architectures, including CNNs, RNNs, LSTMs, BiLSTMs, and multimodal fusion techniques like MCB and late fusion.
• Use a survey-based methodology to synthesize findings from recent literature (2018–2021), focusing on trends, challenges, and open issues in each domain.

Experimental results

Research questions

• RQ1 How do recent MMDL models effectively integrate and represent information across diverse modalities such as image, text, audio, and physiological signals?
• RQ2 What are the key architectural innovations and fusion strategies that have improved performance in multimodal tasks like visual question answering, speech recognition, and emotion detection?
• RQ3 What are the major limitations and open challenges in current MMDL applications, particularly concerning data heterogeneity, dimensionality, and signal preprocessing?
• RQ4 How do existing datasets and evaluation metrics support or constrain progress in multimodal learning across different application domains?
• RQ5 What future research directions are most promising for advancing robust, generalizable, and efficient multimodal deep learning systems?

Key findings

• Multimodal learning significantly enhances performance over unimodal approaches by leveraging complementary information from multiple sensory inputs.
• Attention mechanisms and multimodal transformers have emerged as dominant architectures, enabling effective cross-modal alignment and feature fusion.
• The curse of dimensionality remains a critical challenge in multimodal fusion, particularly when concatenating high-dimensional features from diverse modalities.
• Physiological signal processing for emotion recognition is still in early stages, with preprocessing complexity and limited representative datasets posing major hurdles.
• End-to-end (E2E) learning models show greater versatility in capturing heterogeneous data correlations compared to traditional pipeline-based methods.
• Future research should prioritize multi-platform event detection using transfer learning and improved feature learning via GANs or RNN extensions to enhance accuracy on social media data.