[Paper Review] One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code

Yong Dai, Duyu Tang+8 more|arXiv (Cornell University)|May 12, 2022

Multimodal Machine Learning Applications7 citations

TL;DR

This paper proposes SkillNet, a single multimodal model that sparsely activates task-specific parameter heads for text, image, sound, video, and code. By specializing parameters per modality and activating only relevant parts during inference, SkillNet achieves performance on par with modality-specific models while enabling efficient self-supervised pretraining, outperforming modality-specific baselines on image, video, and code tasks and achieving state-of-the-art accuracy in Chinese text-to-image retrieval with fewer activated parameters.

ABSTRACT

People perceive the world with multiple senses (e.g., through hearing sounds, reading words and seeing objects). However, most existing AI systems only process an individual modality. This paper presents an approach that excels at handling multiple modalities of information with a single model. In our "{SkillNet}" model, different parts of the parameters are specialized for processing different modalities. Unlike traditional dense models that always activate all the model parameters, our model sparsely activates parts of the parameters whose skills are relevant to the task. Such model design enables SkillNet to learn skills in a more interpretable way. We develop our model for five modalities including text, image, sound, video and code. Results show that, SkillNet performs comparably to five modality-specific fine-tuned models. Moreover, our model supports self-supervised pretraining with the same sparsely activated way, resulting in better initialized parameters for different modalities. We find that pretraining significantly improves the performance of SkillNet on five modalities, on par with or even better than baselines with modality-specific pretraining. On the task of Chinese text-to-image retrieval, our final system achieves higher accuracy than existing leading systems including Wukong{ViT-B} and Wenlan 2.0 while using less number of activated parameters.

Motivation & Objective

• To develop a unified model that handles multiple modalities without requiring separate models per modality.
• To enable efficient, interpretable parameter specialization where only relevant model parts are activated per task.
• To investigate whether sparse parameter activation improves pretraining efficiency and downstream performance across diverse modalities.
• To evaluate whether a single model can outperform or match modality-specific models in accuracy while reducing computational cost.

Proposed method

• SkillNet uses a sparse parameter activation mechanism where different parameter heads are specialized for distinct modalities (e.g., text, image, sound, video, code).
• During inference, only the parameter heads relevant to the input modality are activated, while others are deactivated, reducing computational load.
• The model extends the Transformer architecture by generating modality-specific query, key, and value vectors through specialized heads before multi-head attention.
• The model supports self-supervised pretraining using the same sparse activation pattern, enabling better initialization across modalities.
• Pretraining is applied jointly across all modalities, with shared objectives that allow knowledge transfer between tasks.
• The architecture is modality-agnostic and task-agnostic, allowing extension to new modalities with minimal reconfiguration.

Experimental results

Research questions

• RQ1 Can a single model achieve performance comparable to five modality-specific models across text, image, sound, video, and code tasks?
• RQ2 Does sparsely activated parameter routing improve model efficiency and performance compared to dense models?
• RQ3 Can self-supervised pretraining with sparse activation lead to better initialization than modality-specific pretraining?
• RQ4 Does the model generalize well across diverse modalities, including low-resource tasks like text-to-video retrieval?

Key findings

• SkillNet achieves performance comparable to five modality-specific fine-tuned models on all five tasks: text classification, automatic speech recognition, text-to-image retrieval, text-to-video retrieval, and text-to-code retrieval.
• After pretraining, SkillNet outperforms modality-specific pretraining baselines on image, video, and code tasks, achieving 73.59% accuracy on text-to-image retrieval and 81.77% Recall@10 on text-to-video retrieval.
• On Chinese text-to-image retrieval, SkillNet achieves 37.0% Recall@1 on the test set with only 124M activated parameters, outperforming Wukong ${}_{\text{ViT-B}}$ (36.7% R@1, 197M params) and Wenlan 2.0 (34.1% R@1, 445M params).
• Pretraining significantly improves performance across all modalities, with gains exceeding those of modality-specific pretraining on three of five tasks.
• The sparse activation mechanism enables efficient inference, as only a fraction of parameters are used per task, reducing computational cost without sacrificing accuracy.