[Paper Review] Building a mixed-lingual neural TTS system with only monolingual data

Liumeng Xue, Wei Song+3 more|arXiv (Cornell University)|Apr 12, 2019

Natural Language Processing Techniques25 references4 citations

TL;DR

This paper proposes a mixed-lingual neural TTS system for Mandarin-English code-switching using only monolingual data from the target speaker. It leverages an average voice model with speaker embedding in the decoder and phoneme-informed attention to achieve speaker consistency and naturalness, demonstrating that monolingual data can yield high-quality mixed-lingual synthesis when combined with strategic modeling of phonetic and speaker representations.

ABSTRACT

When deploying a Chinese neural text-to-speech (TTS) synthesis system, one of the challenges is to synthesize Chinese utterances with English phrases or words embedded. This paper looks into the problem in the encoder-decoder framework when only monolingual data from a target speaker is available. Specifically, we view the problem from two aspects: speaker consistency within an utterance and naturalness. We start the investigation with an Average Voice Model which is built from multi-speaker monolingual data, i.e. Mandarin and English data. On the basis of that, we look into speaker embedding for speaker consistency within an utterance and phoneme embedding for naturalness and intelligibility and study the choice of data for model training. We report the findings and discuss the challenges to build a mixed-lingual TTS system with only monolingual data.

Motivation & Objective

• To investigate whether a mixed-lingual TTS system can be built using only monolingual data from a single target speaker.
• To evaluate the effectiveness of speaker embedding in maintaining speaker consistency during language switching in utterances.
• To examine whether phoneme embeddings improve naturalness and intelligibility in code-switched speech.
• To determine whether monolingual training data is sufficient for high-quality mixed-lingual synthesis or if mixed-lingual data is essential.
• To analyze the impact of model architecture choices, such as speaker embedding placement and attention mechanisms, on synthesis quality.

Proposed method

• An average voice model (AVM) is trained on multi-speaker monolingual data from Mandarin and English to establish a shared acoustic space.
• Speaker embedding is integrated into the decoder input (SE-DEC) to better control speaker identity and improve consistency during language switching.
• Phoneme-informed attention is implemented via an additional context vector (PECV) and a residual encoder (RES) to enhance alignment and context representation.
• The model is trained on three data configurations: monolingual Mandarin (CORPUS-MAN), monolingual English (CORPUS-ENG), and mixed-lingual (CORPUS-MIX) utterances from the target speaker.
• Performance is evaluated through listening tests comparing AB preference for naturalness and speaker consistency across three languages.
• The study compares SE-DEC with Retrain-AVM and analyzes the impact of training data composition on model generalization.

Experimental results

Research questions

• RQ1 Can the encoder in an encoder-decoder model learn meaningful and distinct phonetic representations for Mandarin and English phonemes when trained only on monolingual data?
• RQ2 How does the placement of speaker embedding (encoder vs. decoder) affect speaker consistency in mixed-lingual utterances?
• RQ3 Can phoneme embeddings improve attention alignment and enhance naturalness and intelligibility in code-switched speech?
• RQ4 Is monolingual data sufficient to train a robust mixed-lingual TTS system, or is mixed-lingual data necessary for optimal performance?
• RQ5 How does the choice of training data (Mandarin-only, English-only, or mixed) influence the model’s ability to generalize across languages?

Key findings

• The average voice model trained on multi-speaker monolingual data learns meaningful phonetic representations and distinguishes between Mandarin and English phonemes, even without bilingual data.
• Speaker embedding integrated into the decoder input (SE-DEC) outperforms encoder-based integration in maintaining speaker consistency during language switching.
• Mixed-lingual training data (CORPUS-MIX) is preferred over monolingual data for mixed-lingual synthesis, though monolingual data still yields acceptable results.
• The residual encoder (SE-DEC-RES) achieves better naturalness than the additional context vector (SE-DEC-PECV) for all three languages, with comparable speaker consistency.
• Listening tests show that models trained on Mandarin data alone are preferred for monolingual Mandarin synthesis, while Mandarin data also outperforms mixed-lingual data for monolingual English synthesis when no English data is available.
• Phoneme-informed attention significantly improves both naturalness and speaker consistency in code-switched utterances, particularly reducing tone inaccuracies in Mandarin words adjacent to English words.