[Paper Review] Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training

Yang, J., Lei He|arXiv (Cornell University)|Jan 20, 2022

Speech Recognition and Synthesis6 citations

TL;DR

This paper proposes a multi-task learning (MTL) and speaker classifier joint training framework to improve cross-lingual speaker similarity in multilingual text-to-speech (TTS) systems. By jointly training a multilingual transformer TTS model with an x-vector speaker classifier and using an approximate parallel scheduled sampling method, the model achieves significantly higher speaker similarity in both seen and unseen speakers across multiple languages, with objective and subjective evaluations confirming consistent gains.

ABSTRACT

In cross-lingual speech synthesis, the speech in various languages can be synthesized for a monoglot speaker. Normally, only the data of monoglot speakers are available for model training, thus the speaker similarity is relatively low between the synthesized cross-lingual speech and the native language recordings. Based on the multilingual transformer text-to-speech model, this paper studies a multi-task learning framework to improve the cross-lingual speaker similarity. To further improve the speaker similarity, joint training with a speaker classifier is proposed. Here, a scheme similar to parallel scheduled sampling is proposed to train the transformer model efficiently to avoid breaking the parallel training mechanism when introducing joint training. By using multi-task learning and speaker classifier joint training, in subjective and objective evaluations, the cross-lingual speaker similarity can be consistently improved for both the seen and unseen speakers in the training set.

Motivation & Objective

• To address the challenge of low speaker similarity in cross-lingual text-to-speech synthesis when only monolingual speaker data is available.
• To improve speaker similarity between synthesized cross-lingual speech and native recordings, especially for unseen speakers.
• To develop a training framework that enhances speaker representation without requiring polyglot data or speaker encoder-based systems.
• To enable effective model fine-tuning for new, unseen speakers with limited data while preserving high speaker similarity.

Proposed method

• Introduces a multi-task learning (MTL) framework that adds speaker and language identity classifiers to the multilingual transformer TTS model, using cross-entropy losses to encourage proper embedding learning.
• Jointly trains the TTS model with an x-vector speaker classifier to minimize the distance between the x-vectors of original recordings and synthesized cross-lingual speech for the same speaker.
• Employs an approximate parallel scheduled sampling scheme to maintain parallel training efficiency despite the need for inference-mode generation during joint training.
• Uses a fixed-dimension speaker embedding and language embedding within the transformer TTS framework to model speaker and language characteristics.
• Applies language-balanced training during speaker extension to prevent overfitting when fine-tuning on new, low-resource speakers.
• Leverages the x-vector system as a differentiable objective to guide the TTS model toward generating speech that matches the speaker's identity across languages.

Experimental results

Research questions

• RQ1 Can multi-task learning improve speaker similarity in cross-lingual TTS without requiring polyglot data or speaker encoders?
• RQ2 How effective is joint training with an x-vector speaker classifier in enhancing cross-lingual speaker similarity for both seen and unseen speakers?
• RQ3 Can the proposed method maintain training efficiency while enabling joint optimization with inference-mode generation during training?
• RQ4 To what extent does the method improve speaker similarity across multiple languages with varying training data sizes?
• RQ5 Can the model generalize to new, unseen speakers with limited data while preserving high speaker similarity?

Key findings

• The +MTL+Joint model reduces cosine distance by 34% on average in cross-lingual synthesis for seen speakers, with the greatest improvement observed in low-resource languages.
• For a new zh-CN speaker with ~9 minutes of data, the similarity MOS increased from 3.99 ± 0.09 (baseline) to 4.21 ± 0.07, indicating a significant subjective improvement.
• In objective evaluations, the cosine distance for the new zh-CN speaker was reduced from 0.096 to 0.092 across languages, showing consistent gains.
• For a new en-GB speaker with ~5 minutes of data, the cosine distance decreased from 0.088 to 0.083 in the target language and from 0.224 to 0.164 in cross-lingual settings.
• The x-vector cosine distance correlates strongly with subjective similarity MOS, validating its use as an objective metric for speaker similarity.
• The method achieves greater improvement in cross-lingual than intra-lingual speaker similarity, confirming the effectiveness of the cross-lingual loss objective.