[Paper Review] MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Paul Pu Liang, Yiwei Lyu+10 more|PubMed|Jul 15, 2021

Topic ModelingComputer Science160 references22 citations

TL;DR

MULTIBENCH introduces a unified, large-scale benchmark for multimodal representation learning, integrating 15 datasets across 10 modalities and 20 tasks spanning 6 research domains. It enables standardized evaluation of model performance, computational complexity, and robustness to noisy or missing modalities, with standardized implementations of 20 core methods that improve SOTA on 9/15 datasets.

ABSTRACT

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark for multimodal learning spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning spanning innovations in fusion paradigms, optimization objectives, and training approaches. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal machine learning research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized implementations, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

Motivation & Objective

• Address the lack of systematic, large-scale benchmarks that evaluate generalization across diverse domains and modalities in multimodal learning.
• Quantify the trade-offs between model performance, training/inference time, and memory usage to support efficiency-aware research.
• Systematically assess robustness to noisy and missing modalities, reflecting real-world data imperfections.
• Provide a standardized, reproducible pipeline for data loading, model training, and evaluation to accelerate research and ensure comparability.
• Foster community-driven expansion by enabling contributions of new datasets, models, and evaluation protocols through a public GitHub repository.

Proposed method

• Design a modular, extensible benchmark framework that unifies 15 diverse datasets spanning 10 modalities (e.g., video, audio, text, force sensors, time-series) and 20 prediction tasks across 6 research areas (e.g., healthcare, finance, robotics).
• Implement an automated end-to-end pipeline for data loading, model training, evaluation, and logging, ensuring reproducibility and ease of use.
• Integrate standardized metrics for evaluating model generalization across domains and modalities, time and space complexity during training and inference, and robustness under modality corruption or missing data.
• Provide standardized implementations of 20 core multimodal learning methods, including diverse fusion paradigms, optimization objectives, and training strategies.
• Support community contributions via a documented GitHub repository with clear guidelines for adding new datasets, dataloaders, and models, including test scripts and compatibility checks.
• Plan for future extensions to include fairness evaluation, with annotated data for bias detection, fairness-aware algorithms, and bias evaluation metrics in the pipeline.

Experimental results

Research questions

• RQ1 How do multimodal models generalize across diverse domains and modalities beyond vision-language tasks?
• RQ2 What are the trade-offs between model performance, training/inference time, and memory consumption in multimodal learning?
• RQ3 How robust are multimodal models to realistic data imperfections such as noisy or missing modalities?
• RQ4 To what extent can standardized implementations of existing methods improve SOTA performance across a broad range of multimodal benchmarks?
• RQ5 How can community-driven contributions enhance the scalability, diversity, and long-term sustainability of multimodal benchmarks?

Key findings

• The standardized implementation of 20 existing multimodal learning methods achieved state-of-the-art performance on 9 out of 15 datasets in the MULTIBENCH benchmark.
• MULTIBENCH enables holistic evaluation of model performance, computational complexity, and robustness to modality imperfections in a single, unified framework.
• The benchmark supports diverse research areas including healthcare, finance, robotics, and affective computing, which are often underrepresented in prior multimodal benchmarks.
• The automated pipeline significantly reduces the barrier to entry for researchers by standardizing data loading, training, and evaluation workflows.
• The framework is extensible and community-driven, with a public GitHub repository that supports contributions of new datasets, models, and evaluation protocols.
• Future extensions will include fairness-aware evaluation, with plans to add human-annotated data for bias detection, fairness-aware algorithms, and bias evaluation metrics.