[Paper Review] M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

Minheng Ni, Haoyang Huang+7 more|arXiv (Cornell University)|Jun 4, 2020

Multimodal Machine Learning Applications39 references43 citations

TL;DR

M3P learns universal multilingual and multimodal representations by jointly pre-training on multilingual text and image-text data, with a novel Multimodal Code-switched Training (MCT) to align non-English languages with vision. It achieves state-of-the-art multilingual image-text retrieval on Multi30K and MSCOCO while maintaining competitive English performance.

ABSTRACT

We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.

Motivation & Objective

• Learn universal representations that map objects across modalities and languages into a common semantic space.
• Integrate multilingual pre-training and multimodal pre-training into a single multitask framework.
• Explicitly align images with non-English languages using code-switching to mitigate data sparsity for multilingual multimodal tasks.

Proposed method

• Use a Transformer backbone initialized from XLM-R to fuse multilingual text and image regions.
• Introduce three data streams: Multilingual Monomodal Stream, Monolingual Multimodal Stream, and Multimodal Code-switched Stream.
• Pre-train with two objectives: Multilingual Masked Language Modeling (xMLM) and Multimodal Code-switched Training (MCT) consisting of MC-MLM, MC-MRM, and MC-VLM.

Experimental results

Research questions

• RQ1 Can multilingual pre-training and multimodal pre-training be effectively unified into a single multitask pre-training framework for multilingual multimodal tasks?
• RQ2 Does Multimodal Code-switched Training improve non-English multilingual-multimodal transfer, especially in zero-shot and few-shot settings?
• RQ3 How do different pre-training tasks contribute to multilingual multimodal retrieval performance across languages?
• RQ4 What is the impact of the number of languages used in MCT on downstream multilingual retrieval?
• RQ5 How does M3P perform in zero-shot, few-shot, and fully supervised fine-tuning scenarios across English and non-English languages?

Key findings

• M3P achieves state-of-the-art results for non-English languages on multilingual image-text retrieval benchmarks MSCOCO and Multi30K.
• In zero-shot and limited-data settings, Multimodal Code-switched Training (MCT) provides significant gains for non-English languages.
• MC-VLM contributes the largest single-task gain among the MCT components, with substantial improvements in English when combined with multilingual multimodal pre-training.
• Expanding MCT to fine-tuning stages yields large non-English gains, especially when code-switched data is used in both pre-training and fine-tuning.
• Using more languages in MCT can improve some languages but may introduce noise; a balanced selection often yields best results.