[Paper Review] High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Paul Pu Liang, Yiwei Lyu+7 more|arXiv (Cornell University)|Mar 2, 2022

Speech and dialogue systems4 citations

TL;DR

This paper proposes HighMMT, a multimodal transformer model that scales to 10 diverse modalities by quantifying modality and interaction heterogeneity using information-theoretic metrics. By measuring transferable information between modalities and modality pairs, HighMMT enables dynamic parameter sharing, improving performance-efficiency trade-offs and enabling effective cross-modal transfer learning across low-resource and unseen modalities and tasks.

ABSTRACT

Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots. While there has been an explosion of interest in multimodal learning, these methods are focused on a small set of modalities primarily in language, vision, and audio. In order to accelerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. Since adding new models for every new modality becomes prohibitively expensive, a critical technical challenge is heterogeneity quantification: how can we measure which modalities encode similar information and interactions in order to permit parameter sharing with previous modalities? This paper proposes two new information theoretic metrics for heterogeneity quantification: (1) modality heterogeneity studies how similar 2 modalities {X1,X2} are by measuring how much information can be transferred from X1 to X2, while (2) interaction heterogeneity studies how similarly pairs of modalities {X1,X2}, {X3,X4} interact by measuring how much information can be transferred from fusing {X1,X2} to {X3,X4}. We show the importance of these 2 proposed metrics as a way to automatically prioritize the fusion of modalities that contain unique information or interactions. The result is a single model, HighMMT, that scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas. Not only does HighMMT outperform prior methods on the tradeoff between performance and efficiency, it also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning.

Motivation & Objective

• To address the challenge of efficient representation learning in high-modality scenarios involving many diverse modalities beyond traditional language, vision, and audio.
• To quantify heterogeneity in modality similarity and interaction patterns to enable intelligent parameter sharing across similar modalities and fusion patterns.
• To develop a unified model that generalizes across diverse tasks and modalities while maintaining high performance and efficiency.
• To enable effective cross-modal transfer learning, especially in low-resource and partially observable settings.
• To provide a standardized, scalable framework for multimodal representation learning that supports future research and generalization to understudied modalities.

Proposed method

• Proposes two information-theoretic metrics: modality heterogeneity, measuring how much information can be transferred from one modality to another, and interaction heterogeneity, measuring transferability between fused modality pairs.
• Uses these metrics to dynamically group modalities and fusion patterns, enabling selective parameter sharing based on similarity in information content and interaction dynamics.
• Designs a single HighMMT model architecture that shares unimodal and multimodal encoder parameters across modalities and tasks, with modality-specific embeddings for performance gains.
• Employs multitask learning with weighted loss functions across 15 diverse tasks spanning 5 research areas to stabilize training and improve generalization.
• Applies transfer learning by pretraining on source tasks and fine-tuning on new target modalities and tasks, demonstrating strong zero-shot and few-shot transfer capabilities.
• Introduces task-specific loss weighting to handle prediction objectives of different scales (e.g., MSE vs. accuracy), improving training stability and performance.

Experimental results

Research questions

• RQ1 How can we measure the similarity of modalities and their interactions in high-modality settings to enable effective parameter sharing?
• RQ2 Can information-theoretic metrics of modality and interaction heterogeneity guide dynamic parameter sharing in multimodal models?
• RQ3 Does a unified HighMMT model trained on diverse modalities and tasks outperform task-specific models in terms of performance and efficiency?
• RQ4 To what extent can HighMMT generalize to new, unseen modalities and tasks through transfer learning?
• RQ5 How does multitask learning with heterogeneous objectives and partial observability affect model generalization and robustness?

Key findings

• HighMMT achieves superior performance-efficiency trade-offs compared to both task-specific state-of-the-art models and fully shared multimodal models.
• Performance continues to improve with each additional modality added, demonstrating strong scalability and incremental learning benefits.
• The model successfully transfers knowledge to new, previously unseen modalities and tasks during fine-tuning, especially in low-resource settings.
• Cross-modal transfer is significantly enhanced when pretraining includes a broader set of source tasks, with gains most pronounced on low-resource targets.
• Well-tuned task weighting improves training dynamics and reduces overfitting, indicating regularization effects from multitask learning.
• Even without modality-specific embeddings, shared unimodal encoders achieve strong performance, suggesting they learn generalizable, modality-invariant features.