[Paper Review] 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities

Roman Bachmann, Oğuzhan Fatih Kar+7 more|arXiv (Cornell University)|Jun 13, 2024

Quality Function Deployment in Product Design4 citations

TL;DR

This paper introduces 4M-21, a single unified vision model trained on 21 diverse modalities—including images, text, feature maps, human poses, and semantic instances—via modality-specific discrete tokenization. It achieves state-of-the-art performance across tens of tasks without performance degradation, enabling any-to-any generation and co-training on vision and language while scaling to 3 billion parameters.

ABSTRACT

Current multimodal and multitask foundation models like 4M or UnifiedIO show promising results, but in practice their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually rather small) number of modalities and tasks they are trained on. In this paper, we expand upon the capabilities of them by training a single model on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example image metadata or color palettes. A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text. Through this, we expand on the out-of-the-box capabilities of multimodal models and specifically show the possibility of training one model to solve at least 3x more tasks/modalities than existing ones and doing so without a loss in performance. This enables more fine-grained and controllable multimodal generation capabilities and allows us to study the distillation of models trained on diverse data and objectives into a unified model. We successfully scale the training to a three billion parameter model using tens of modalities and different datasets. The resulting models and training code are open sourced at 4m.epfl.ch.

Motivation & Objective

• Address the limited out-of-the-box capability of existing foundation models that support only a small number of modalities and tasks.
• Overcome performance degradation in multitask learning when scaling to highly diverse modalities with varying data types and dimensions.
• Enable fine-grained, controllable multimodal generation by integrating structured and semantic modalities such as human poses, segmentation masks, and color palettes.
• Demonstrate that co-training on vision and language objectives can be effectively unified in a single model without negative transfer.
• Scale the model to 3 billion parameters and 0.5 billion training samples while maintaining strong performance across all tasks.

Proposed method

• Train a single Transformer-based model using multimodal masked modeling, where all inputs and outputs are mapped into a shared discrete token space.
• Apply modality-specific discrete tokenizers: ViT-based VQ-VAEs for image-like data (RGB, edges), MLP-based VAEs for vectors (DINOv2, ImageBind), and WordPiece tokenizers for text-like data (captions, metadata).
• Integrate diverse modalities including RGB images, depth, normals, semantic segmentation, instance masks (SAM), 3D human poses (4DHumans), Canny edges, color palettes, and T5-XXL text embeddings.
• Use co-training on large-scale multimodal datasets (e.g., CC12M) and text corpora to jointly optimize vision and language objectives.
• Optimize tokenization schemes via ablation—evaluating L1, MSE, cross-entropy, and Dice losses—on SAM instances using IoU as the reconstruction metric.
• Scale training to 3B parameters using 128 A100 GPUs for up to 11 days, with dedicated training for 10 auxiliary tokenizers on 8 A100s each.

Experimental results

Research questions

• RQ1 Can a single unified model be trained on tens of highly diverse modalities without performance degradation compared to specialized models?
• RQ2 How does modality-specific discrete tokenization enable effective representation learning across heterogeneous data types like images, vectors, and structured poses?
• RQ3 To what extent does co-training on vision and language objectives improve generalization and zero-shot transfer in a unified model?
• RQ4 What is the optimal trade-off between reconstruction quality and sequence length in tokenizing complex modalities like instance segmentation masks?
• RQ5 Can the model support any-to-any generation across modalities, including novel ones like color palettes and metadata, with high fidelity and controllability?

Key findings

• 4M-21 achieves state-of-the-art performance across 21 diverse modalities—3× more than existing any-to-any models—without performance degradation compared to specialized models.
• The model demonstrates strong zero-shot generalization and controllable generation, including one-to-all generation across all 21 modalities.
• Mask-based tokenization of SAM instances with 16 tokens, 1024 vocabulary size, and 64×64 resolution achieved the best balance of reconstruction quality (IoU) and sequence length.
• Cross-entropy loss in mask tokenization produced smoother, better-covered reconstructions than L1 or MSE losses.
• Training the full model required approximately 120,000 A100-hours, with additional 60 A100-days for tokenizer training, totaling ~150,000 A100-hours for the full project.
• The model supports cross-modal retrieval and generation across modalities such as 3D human poses, image embeddings, and color palettes, enabling new interaction paradigms.