[Paper Review] SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

Paul-Ambroise Duquenne, Hongyu Gong+8 more|arXiv (Cornell University)|Nov 8, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces SpeechMatrix (SM), a large-scale multilingual speech-to-speech translation corpus mined from European Parliament recordings, comprising 136 language pairs and 418,000 hours of aligned speech. The authors demonstrate strong S2ST performance using only mined data, showing that multilingual training, model pre-training, and Mixture-of-Experts sparsity significantly improve translation quality across multiple benchmarks.

ABSTRACT

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available.

Motivation & Objective

• To address the critical data scarcity in speech-to-speech translation (S2ST) by creating a large-scale, freely available multilingual speech-to-speech corpus.
• To evaluate the quality and utility of mined speech alignments across 17 languages and 136 language pairs.
• To explore multilingual S2ST training and demonstrate performance gains from pre-training and sparse model scaling (MoE).
• To enable zero-shot and few-shot transfer learning in multilingual S2ST through shared multilingual representations.

Proposed method

• Leveraged multilingual speech-text sentence embeddings from Duquenne et al. (2021) to mine speech-to-speech alignments across 17 languages in the European Parliament corpus.
• Trained multilingual HuBERT models on 17 languages to extract discrete speech units for downstream S2ST and vocoder training.
• Constructed a multilingual speech-to-speech translation pipeline using sequence-to-sequence models with shared encoder-decoder architectures.
• Applied Mixture-of-Experts (MoE) with GShard and Base Layer sparsity to scale model capacity efficiently without proportional inference cost increase.
• Used pre-trained modules (e.g., HuBERT, mBART) to improve initialization and generalization in low-resource settings.
• Evaluated models on Europarl-ST, VoxPopuli, and FLEURS benchmarks using BLEU scores, with ablation studies on data selection and model architecture.

Experimental results

Research questions

• RQ1 Can large-scale, multilingual speech-to-speech alignments be effectively mined from monolingual speech and text using multilingual embeddings?
• RQ2 How does multilingual S2ST training compare to bilingual training in terms of zero-shot and in-domain performance?
• RQ3 To what extent do model pre-training and Mixture-of-Experts sparsity improve S2ST performance on mined data?
• RQ4 Does the quality of mined alignments support strong S2ST performance without human-annotated parallel data?

Key findings

• SpeechMatrix contains 418,000 hours of speech across 136 language pairs, making it the largest freely available multilingual S2ST corpus to date.
• Bilingual S2ST models trained solely on mined data achieve strong performance, with +3.8 BLEU average gain on Europarl-ST and VoxPopuli compared to baseline models.
• Multilingual training with the XM Transformer model yields +7.9 BLEU gains on Europarl-ST and +5.1 BLEU on FLEURS over bilingual training.
• MoE-based sparse scaling with GShard (64 experts) improves performance by +1.0 BLEU on average over dense models in the Europarl-ST set, with larger gains in-domain.
• The all-to-English multilingual model with MoE-GShard (4.3B parameters) achieves +0.9 BLEU on Europarl-ST and +0.2 BLEU on FLEURS compared to a dense 1.2B model with same FLOPs.
• Performance degrades slightly on out-of-domain FLEURS when expanding to all 17 languages, indicating domain shift challenges in multilingual generalization.