[Paper Review] i-Code: An Integrative and Composable Multimodal Learning Framework

Ziyi Yang, Yuwei Fang+18 more|arXiv (Cornell University)|May 3, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

i-Code is a self-supervised, integrative multimodal framework that combines vision, speech, and language representations using pretrained single-modality encoders and a novel fusion network with cross-modality attention. It achieves state-of-the-art performance on five video understanding tasks and the GLUE benchmark, improving by up to 11% over prior methods through joint pretraining on dual- and triple-modality data with masked unit modeling and contrastive learning objectives.

ABSTRACT

Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview. Most current pretraining methods, however, are limited to one or two modalities. We present i-Code, a self-supervised pretraining framework where users may flexibly combine the modalities of vision, speech, and language into unified and general-purpose vector representations. In this framework, data from each modality are first given to pretrained single-modality encoders. The encoder outputs are then integrated with a multimodal fusion network, which uses novel attention mechanisms and other architectural innovations to effectively combine information from the different modalities. The entire system is pretrained end-to-end with new objectives including masked modality unit modeling and cross-modality contrastive learning. Unlike previous research using only video for pretraining, the i-Code framework can dynamically process single, dual, and triple-modality data during training and inference, flexibly projecting different combinations of modalities into a single representation space. Experimental results demonstrate how i-Code can outperform state-of-the-art techniques on five video understanding tasks and the GLUE NLP benchmark, improving by as much as 11% and demonstrating the power of integrative multimodal pretraining.

Motivation & Objective

• To address the challenge of training effective three-modality models (vision, language, speech) given the scarcity of large-scale, fully annotated triple-modality data.
• To enable flexible, dynamic processing of single, dual, or triple modalities within a unified representation space.
• To improve multimodal representation learning by leveraging large-scale dual-modality datasets (e.g., image-caption, speech-transcript, video-narration) alongside triple-modality data.
• To close the performance gap between multimodal models and single-modality language models on NLP benchmarks by enhancing language understanding through cross-modal pretraining.
• To design a fusing architecture that effectively integrates outputs from state-of-the-art single-modality encoders using novel attention mechanisms and self-supervised objectives.

Proposed method

• Utilizes pretrained single-modality encoders (vision, speech, language) as feature extractors, avoiding training from scratch.
• Employs a multimodal fusion network with cross-attention mechanisms that allow interaction between different modality embeddings.
• Applies masked unit modeling: treats modality inputs as discrete tokens and predicts masked tokens across modalities during pretraining.
• Introduces cross-modality contrastive learning: determines whether two input signals (e.g., image and text) originate from the same sample pair or triplet.
• Pretrains end-to-end on diverse data including video (V+L+S), image-caption (V+L), speech-transcript (S+L), and video narration (V+S) to increase data efficiency.
• Uses a shared representation space where outputs from different modality combinations are projected into a unified embedding space.

Experimental results

Research questions

• RQ1 Can a multimodal pretraining framework effectively learn joint representations across vision, language, and speech using limited triple-modality data?
• RQ2 Does pretraining on dual-modality data (e.g., V+L, S+L, V+S) yield better performance than relying solely on triple-modality video data?
• RQ3 Can a fusing module with novel attention mechanisms outperform standard fusion strategies in integrating heterogeneous modality representations?
• RQ4 To what extent can a multimodal model match or exceed the performance of single-modality language models on NLP benchmarks like GLUE?
• RQ5 How does the integration of masked unit modeling and contrastive learning improve cross-modal alignment and downstream task generalization?

Key findings

• i-Code achieves state-of-the-art performance on five video understanding benchmarks, outperforming prior methods by up to 11% in some cases.
• On the GLUE NLP benchmark, i-Code achieves an average score of 89.0, surpassing DeBERTa V3 Base (88.6) and significantly outperforming other multimodal models like UNITER (72.9) and CLIP (50.5).
• The model improves performance on language-only tasks by 1.4–4.5 points over previous multimodal models, closing the gap with dedicated language models.
• Language-speech (L+S) combination yields the best dual-modality performance on MOSEI emotion recognition (F1: 56.1), outperforming single modalities.
• Pretraining on dual-modality data (i-Code D+M) achieves performance comparable to or better than pretraining on triple-modality video data (i-Code V+M), demonstrating data efficiency.
• The ablation study confirms that all three modalities contribute to improved performance, with V+L+S achieving the highest accuracy (49.4%) on MOSEI emotion recognition.