[论文解读] M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

Minheng Ni, Haoyang Huang+7 more|arXiv (Cornell University)|Jun 4, 2020

Multimodal Machine Learning Applications参考文献 39被引用 43

一句话总结

M3P 通过在多语言文本和图像文本数据上联合预训练来学习通用的多语言和多模态表示，提出新的 Multimodal Code-switched Training (MCT) 来将非英语语言与视觉对齐。它在 Multi30K 和 MSCOCO 上实现了最先进的多语言图像文本检索，同时保持竞争力的英语表现。

ABSTRACT

We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.

研究动机与目标

• 学习将跨模态与跨语言的对象映射到一个公共语义空间的通用表示。
• 将多语言预训练和多模态预训练整合到一个单一的多任务框架中。
• 通过使用代码切换显式将图像与非英语语言对齐，以缓解多语言多模态任务的数据稀缺问题。

提出的方法

• 使用从 XLM-R 初始化的 Transformer 主干来融合多语言文本和图像区域。
• 引入三条数据流：Multilingual Monomodal Stream、Monolingual Multimodal Stream 和 Multimodal Code-switched Stream。
• 使用两个目标进行预训练：Multilingual Masked Language Modeling (xMLM) 和 Multimodal Code-switched Training (MCT)，包括 MC-MLM、MC-MRM 和 MC-VLM。

实验结果

研究问题

• RQ1 Can multilingual pre-training and multimodal pre-training be effectively unified into a single multitask pre-training framework for multilingual multimodal tasks?
• RQ2 Does Multimodal Code-switched Training improve non-English multilingual-multimodal transfer, especially in zero-shot and few-shot settings?
• RQ3 How do different pre-training tasks contribute to multilingual multimodal retrieval performance across languages?
• RQ4 What is the impact of the number of languages used in MCT on downstream multilingual retrieval?
• RQ5 How does M3P perform in zero-shot, few-shot, and fully supervised fine-tuning scenarios across English and non-English languages?

主要发现

• M3P achieves state-of-the-art results for non-English languages on multilingual image-text retrieval benchmarks MSCOCO and Multi30K.
• In zero-shot and limited-data settings, Multimodal Code-switched Training (MCT) provides significant gains for non-English languages.
• MC-VLM contributes the largest single-task gain among the MCT components, with substantial improvements in English when combined with multilingual multimodal pre-training.
• Expanding MCT to fine-tuning stages yields large non-English gains, especially when code-switched data is used in both pre-training and fine-tuning.
• Using more languages in MCT can improve some languages but may introduce noise; a balanced selection often yields best results.