[Paper Review] MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

Yatai Ji, Junjie Wang+7 more|arXiv (Cornell University)|Oct 11, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes MAP, a multimodal vision-language pre-training model that models uncertainty by representing modality features as Gaussian distributions via a Probability Distribution Encoder (PDE). By integrating uncertainty-aware pre-training tasks—D-VLC, D-MLM, and D-ITM—MAP achieves state-of-the-art performance on downstream tasks like image-text retrieval, VQA, and visual reasoning, while enabling diverse, uncertainty-driven predictions through distribution sampling.

ABSTRACT

Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets. Such uncertainty is problematic for our interpretation, including inter- and intra-modal uncertainty. Little effort has studied the modeling of this uncertainty, particularly in pre-training on unlabeled datasets and fine-tuning in task-specific downstream datasets. In this paper, we project the representations of all modalities as probabilistic distributions via a Probability Distribution Encoder (PDE) by utilizing sequence-level interactions. Compared to the existing deterministic methods, such uncertainty modeling can convey richer multimodal semantic information and more complex relationships. Furthermore, we integrate uncertainty modeling with popular pre-training frameworks and propose suitable pre-training tasks: Distribution-based Vision-Language Contrastive learning (D-VLC), Distribution-based Masked Language Modeling (D-MLM), and Distribution-based Image-Text Matching (D-ITM). The fine-tuned models are applied to challenging downstream tasks, including image-text retrieval, visual question answering, visual reasoning, and visual entailment, and achieve state-of-the-art results.

Motivation & Objective

• To address the challenge of semantic uncertainty in multimodal understanding, particularly inter- and intra-modal ambiguity in vision-language tasks.
• To model uncertainty in multimodal representations not as deterministic points but as probability distributions, capturing richer semantic relationships.
• To develop novel pre-training tasks that leverage distributional representations for large-scale unlabeled data.
• To integrate uncertainty modeling into a unified, end-to-end pre-training framework for improved downstream performance.
• To enable diverse, plausible predictions in tasks like VQA and image captioning through sampling from learned distributions.

Proposed method

• The Probability Distribution Encoder (PDE) transforms sequence-level representations from vision and language modalities into Gaussian distributions using attention-based interactions between tokens and image patches.
• Each modality's representation is parameterized as a mean and variance vector, forming a multivariate Gaussian distribution to encode uncertainty.
• Three new pre-training tasks are introduced: Distribution-based Vision-Language Contrastive learning (D-VLC), Distribution-based Masked Language Modeling (D-MLM), and Distribution-based Image-Text Matching (D-ITM).
• D-VLC aligns entire distribution representations across modalities for coarse-grained cross-modal understanding.
• D-MLM and D-ITM refine alignment at the token and overall levels, respectively, using distributional predictions.
• The model is fine-tuned end-to-end on downstream tasks such as VQA, image retrieval, and visual entailment.

Experimental results

Research questions

• RQ1 Can modeling multimodal representations as probability distributions improve semantic understanding in vision-language tasks?
• RQ2 How does uncertainty-aware pre-training with distributional representations affect downstream performance on vision-language benchmarks?
• RQ3 To what extent do distribution representations enable diverse and plausible predictions in tasks like VQA and captioning?
• RQ4 Which pre-training components (D-VLC, D-MLM, D-ITM) contribute most significantly to model performance?
• RQ5 Does the integration of uncertainty modeling alleviate ambiguity in ambiguous visual and linguistic contexts?

Key findings

• MAP achieves state-of-the-art performance on image-text retrieval, VQA2.0, NLVR2, and SNLI-VE, demonstrating the effectiveness of uncertainty-aware pre-training.
• The D-MLM pre-training task contributes the most to performance, with models trained without it showing the worst results across all strategies.
• Pre-training enables deeper models (e.g., 8 layers) to generalize better, overcoming performance bottlenecks seen in randomly initialized models.
• Visualization shows that distribution representations cluster semantically similar images and captions together, with overlapping ellipses indicating shared meaning.
• Sampling from distribution representations generates diverse plausible answers (e.g., 'field', 'park', 'grass' for the same image), unlike deterministic models that produce only one answer.
• The PDE module enables the model to express complex relationships and uncertainty, improving robustness in ambiguous scenarios.