[Paper Review] Transfer Learning from Monolingual ASR to Transcription-free Cross-lingual Voice Conversion

Che‐Jui Chang|arXiv (Cornell University)|Sep 30, 2020

Speech Recognition and Synthesis28 references4 citations

TL;DR

This paper proposes a transcription-free cross-lingual voice conversion method that leverages transfer learning from a monolingual automatic speech recognition (ASR) model to align content across different languages. By extracting phonetic features from source-language speech using a pre-trained ASR model and training a Seq2Seq model to predict target mel-spectrograms, the approach achieves speaker-dependent voice conversion without parallel data or transcriptions, attaining MOS scores of 3.83 (quality) and 3.54 (similarity) on the VCC 2020 dataset.

ABSTRACT

Cross-lingual voice conversion (VC) is a task that aims to synthesize target voices with the same content while source and target speakers speak in different languages. Its challenge lies in the fact that the source and target data are naturally non-parallel, and it is even difficult to bridge the gaps between languages with no transcriptions provided. In this paper, we focus on knowledge transfer from monolin-gual ASR to cross-lingual VC, in order to address the con-tent mismatch problem. To achieve this, we first train a monolingual acoustic model for the source language, use it to extract phonetic features for all the speech in the VC dataset, and then train a Seq2Seq conversion model to pre-dict the mel-spectrograms. We successfully address cross-lingual VC without any transcription or language-specific knowledge for foreign speech. We experiment this on Voice Conversion Challenge 2020 datasets and show that our speaker-dependent conversion model outperforms the zero-shot baseline, achieving MOS of 3.83 and 3.54 in speech quality and speaker similarity for cross-lingual conversion. When compared to Cascade ASR-TTS method, our proposed one significantly reduces the MOS drop be-tween intra- and cross-lingual conversion.

Motivation & Objective

• To address the challenge of content mismatch in cross-lingual voice conversion when source and target languages are non-parallel and no transcriptions are available.
• To explore knowledge transfer from monolingual ASR models to improve alignment and synthesis in zero-shot cross-lingual voice conversion.
• To reduce the performance gap between intra- and cross-lingual voice conversion without relying on language-specific or transcription-based supervision.
• To develop a speaker-dependent voice conversion system that generalizes across languages using only acoustic features from a source-language ASR model.

Proposed method

• Train a monolingual ASR model on the source language to extract phonetic embeddings from all speech in the dataset.
• Use the trained ASR model to generate content-aligned phonetic representations for both source and target speech, enabling cross-lingual alignment.
• Feed the extracted phonetic features into a Seq2Seq architecture to predict target speaker's mel-spectrograms directly from source speech.
• Train the Seq2Seq model end-to-end using only acoustic features and target speaker embeddings, without requiring transcriptions or language-specific models.
• Apply speaker-dependent conditioning during inference to preserve target speaker identity in the synthesized speech.
• Utilize the VCC 2020 dataset for evaluation, comparing performance against zero-shot and cascade ASR-TTS baselines.

Experimental results

Research questions

• RQ1 Can a monolingual ASR model be effectively used to transfer content representation across languages in voice conversion without transcriptions?
• RQ2 How does knowledge transfer from ASR improve cross-lingual voice conversion performance compared to zero-shot or cascade methods?
• RQ3 To what extent can phonetic features extracted from a source-language ASR model enable content-preserving voice conversion in a target language?
• RQ4 Does the proposed method reduce the MOS drop between intra- and cross-lingual conversion compared to existing approaches?

Key findings

• The proposed method achieves a mean opinion score (MOS) of 3.83 for speech quality and 3.54 for speaker similarity in cross-lingual voice conversion on the VCC 2020 dataset.
• The model outperforms the zero-shot baseline in both speech quality and speaker similarity, demonstrating the effectiveness of monolingual ASR transfer.
• The MOS drop between intra- and cross-lingual conversion is significantly reduced compared to the cascade ASR-TTS method, indicating better generalization.
• The approach successfully enables speaker-dependent voice conversion without requiring parallel data, transcriptions, or language-specific models.
• Phonetic features extracted from a source-language ASR model provide sufficient content representation for cross-lingual alignment.