[Paper Review] mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset

Luiz Bonifacio, Jeronymo, Vitor+5 more|arXiv (Cornell University)|Aug 31, 2021

Topic ModelingComputer Science43 references18 citations

TL;DR

This paper introduces mMARCO, a multilingual adaptation of the MS MARCO passage ranking dataset translated into 13 languages using machine translation. The authors demonstrate that finetuning multilingual models on mMARCO improves zero-shot performance on Mr. TyDi, and that translation quality correlates with retrieval effectiveness, validating the utility of high-quality translations for multilingual IR.

ABSTRACT

The MS MARCO ranking dataset has been widely used for training deep learning models for IR tasks, achieving considerable effectiveness on diverse zero-shot scenarios. However, this type of resource is scarce in languages other than English. In this work, we present mMARCO, a multilingual version of the MS MARCO passage ranking dataset comprising 13 languages that was created using machine translation. We evaluated mMARCO by finetuning monolingual and multilingual reranking models, as well as a multilingual dense retrieval model on this dataset. We also evaluated models finetuned using the mMARCO dataset in a zero-shot scenario on Mr. TyDi dataset, demonstrating that multilingual models finetuned on our translated dataset achieve superior effectiveness to models finetuned on the original English version alone. Our experiments also show that a distilled multilingual reranker is competitive with non-distilled models while having 5.4 times fewer parameters. Lastly, we show a positive correlation between translation quality and retrieval effectiveness, providing evidence that improvements in translation methods might lead to improvements in multilingual information retrieval. The translated datasets and finetuned models are available at https://github.com/unicamp-dl/mMARCO.

Motivation & Objective

• Address the scarcity of multilingual training and evaluation data for neural IR models beyond English.
• Create a high-quality, multilingual passage ranking dataset by translating the original MS MARCO dataset into 13 languages using machine translation.
• Evaluate the effectiveness of monolingual and multilingual models finetuned on the translated dataset in both zero-shot and in-domain settings.
• Investigate the impact of translation quality on downstream retrieval performance to validate the reliability of translated data.
• Provide a benchmark for multilingual IR by releasing the mMARCO dataset and finetuned models for public use.

Proposed method

• Translate the MS MARCO training and evaluation sets into 13 languages (e.g., Portuguese, Spanish, German, Chinese, Russian) using neural machine translation models.
• Use the Helsinki-NLP translation models for translation, with BLEU scores used to assess translation quality on the Tatoeba dataset.
• Fine-tune monolingual and multilingual Transformer-based models (e.g., mMiniLM, mT5, mBERT) on the translated mMARCO datasets for both dense retrieval and reranking tasks.
• Evaluate finetuned models on the Mr. TyDi benchmark in a zero-shot cross-lingual setting to assess transferability.
• Train a distilled multilingual reranker (mMiniLM) to compare efficiency and performance against larger models.
• Analyze the correlation between translation quality (BLEU) and retrieval effectiveness (MRR@10) across languages.

Experimental results

Research questions

• RQ1 To what extent can machine-translated MS MARCO data support effective finetuning of multilingual IR models?
• RQ2 Does finetuning multilingual models on mMARCO improve zero-shot performance on unseen languages compared to monolingual finetuning?
• RQ3 How does translation quality, measured by BLEU, correlate with retrieval effectiveness in multilingual IR?
• RQ4 Can distilled multilingual models achieve competitive performance with larger models when finetuned on mMARCO?
• RQ5 Does multilingual finetuning on mMARCO yield better performance than monolingual finetuning, especially for low-resource languages?

Key findings

• Multilingual models finetuned on mMARCO outperform models finetuned only on the original English MS MARCO dataset in zero-shot evaluation on the Mr. TyDi benchmark.
• The mT5 model finetuned on multilingual mMARCO (including English and Portuguese) achieved an MRR@10 of 0.728 on the Portuguese MS MARCO subset, outperforming monolingual models.
• A distilled mMiniLM reranker achieved competitive performance with 5.4 times fewer parameters than its non-distilled counterpart.
• There is a weak but positive correlation (R² ≈ 0.33) between translation quality (BLEU) and retrieval effectiveness (MRR@10), indicating that better translations lead to better IR results.
• For non-Latin scripts (e.g., Chinese, Russian), commercial translation models outperformed open-source models, suggesting language-specific translation quality impacts IR performance.
• Multilingual finetuning on mMARCO led to models that performed nearly as well as monolingual models on both English and Portuguese, demonstrating the robustness of the translated data.