[Paper Review] Multilingual Speech-to-Speech Translation into Multiple Target Languages

Hongyu Gong, Ning Dong+4 more|arXiv (Cornell University)|Jul 17, 2023

Speech Recognition and Synthesis4 citations

TL;DR

This paper presents the first multilingual speech-to-speech translation system supporting multiple target languages, using a speech-to-masked-unit (S2MU) model with language-specific unit masking and a multilingual vocoder with language embedding and auxiliary language identification loss. The approach achieves +5.2 BLEU average gain over bilingual models on in-domain data and +2.7 on out-of-domain data when translating English into 16 target languages.

ABSTRACT

Speech-to-speech translation (S2ST) enables spoken communication between people talking in different languages. Despite a few studies on multilingual S2ST, their focus is the multilinguality on the source side, i.e., the translation from multiple source languages to one target language. We present the first work on multilingual S2ST supporting multiple target languages. Leveraging recent advance in direct S2ST with speech-to-unit and vocoder, we equip these key components with multilingual capability. Speech-to-masked-unit (S2MU) is the multilingual extension of S2U, which applies masking to units which don't belong to the given target language to reduce the language interference. We also propose multilingual vocoder which is trained with language embedding and the auxiliary loss of language identification. On benchmark translation testsets, our proposed multilingual model shows superior performance than bilingual models in the translation from English into $16$ target languages.

Motivation & Objective

• To enable multilingual speech-to-speech translation with support for multiple target languages, moving beyond prior work focused only on multilingual sources to a single target.
• To address the challenge of language interference and increased unit vocabulary size when combining multiple languages in a single S2ST model.
• To improve scalability and performance by introducing a multilingual vocoder with language embedding and auxiliary language identification loss.
• To enable knowledge transfer across languages through shared multilingual modeling, especially benefiting low-resource languages.

Proposed method

• Propose a speech-to-masked-unit (S2MU) model that applies masking to units not belonging to the target language, reducing cross-lingual interference during unit prediction.
• Introduce a multilingual vocoder trained with language embeddings and an auxiliary loss for language identification to improve generalization across target languages.
• Use pre-trained HuBERT models to extract discrete units from target speech, with unit dictionaries concatenated across languages for multilingual modeling.
• Train the S2U model end-to-end with masked units to focus on relevant language-specific units during translation.
• Apply a two-stage training pipeline: first train S2MU with masked units, then train the multilingual vocoder using predicted units and language identifiers.
• Use ASR-BLEU as evaluation metric, where generated speech is transcribed by a pre-trained ASR model and compared to reference texts.

Figure 1: Architecture of multilingual speech-to-speech translation model.

Experimental results

Research questions

• RQ1 Can a single multilingual model effectively support speech-to-speech translation into multiple target languages, rather than just multiple source languages?
• RQ2 How can language interference be mitigated when combining multiple language unit vocabularies in a shared S2U model?
• RQ3 Can a multilingual vocoder trained with language embeddings and auxiliary language identification improve speech synthesis quality and translation performance across diverse target languages?
• RQ4 Does knowledge transfer across languages in multilingual S2ST lead to performance gains, especially in low-resource settings?
• RQ5 How does the choice of model capacity (e.g., S2U vs. Textless) interact with language resource availability to affect translation performance?

Key findings

• The proposed multilingual S2ST model achieves an average BLEU gain of +5.2 over bilingual S2U models with monolingual vocoders on in-domain test sets across 16 target languages.
• On out-of-domain data, the multilingual model achieves an average BLEU improvement of +2.7 over bilingual baselines, demonstrating robustness to domain shift.
• The S2MU model with multilingual vocoder outperforms bilingual Textless models by +1.1 and +1.6 BLEU on in-domain and out-of-domain sets, respectively.
• Multilingual vocoders improve translation performance by reducing WER in resynthesis, with gains most pronounced in high-resource directions such as en-es, where BLEU increases by +2.0 with S2MU.
• The performance of multilingual models is strongly correlated with S2MU inference quality; languages like Slovak (sk) see WER reduction of 5% but limited BLEU gains due to low-quality units.
• In low-resource directions such as et, fi, and lt, all models perform poorly, indicating data scarcity remains a major bottleneck despite multilingual training.

Figure 2: Architecture of multilingual vocoder.