[Paper Review] Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis

Jia Ye, Yu Zhang+9 more|arXiv (Cornell University)|Jun 12, 2018

Speech Recognition and SynthesisComputer Science21 references433 citations

TL;DR

The paper presents a multispeaker TTS system that decouples speaker representation from synthesis, using a pre-trained speaker encoder to enable zero-shot synthesis of unseen speakers via a Tacotron 2–based synthesizer and WaveNet vocoder. It shows that a large, diverse, unlabeled speaker dataset improves generalization to new voices.

ABSTRACT

We describe a neural network-based system for text-to-speech (TTS) synthesis that is able to generate speech audio in the voice of many different speakers, including those unseen during training. Our system consists of three independently trained components: (1) a speaker encoder network, trained on a speaker verification task using an independent dataset of noisy speech from thousands of speakers without transcripts, to generate a fixed-dimensional embedding vector from seconds of reference speech from a target speaker; (2) a sequence-to-sequence synthesis network based on Tacotron 2, which generates a mel spectrogram from text, conditioned on the speaker embedding; (3) an auto-regressive WaveNet-based vocoder that converts the mel spectrogram into a sequence of time domain waveform samples. We demonstrate that the proposed model is able to transfer the knowledge of speaker variability learned by the discriminatively-trained speaker encoder to the new task, and is able to synthesize natural speech from speakers that were not seen during training. We quantify the importance of training the speaker encoder on a large and diverse speaker set in order to obtain the best generalization performance. Finally, we show that randomly sampled speaker embeddings can be used to synthesize speech in the voice of novel speakers dissimilar from those used in training, indicating that the model has learned a high quality speaker representation.

Motivation & Objective

• Learn a speaker-encoder that captures speaker characteristics from short, untranscribed reference audio.
• Train a multispeaker TTS synthesizer conditioned on the speaker embedding without transcripts for the synthesizer data.
• Demonstrate zero-shot voice transfer to speakers unseen during training.
• Evaluate naturalness and speaker similarity using MOS and speaker-verification metrics.
• Analyze how speaker encoder training data size and diversity affect transfer quality.

Proposed method

• Independent training of three components: a speaker encoder trained on speaker verification with untranscribed, noisy data; a Tacotron 2–based sequence-to-sequence synthesizer conditioned on the speaker embedding; and a WaveNet vocoder converting mel spectrograms to waveform.
• The speaker embedding is obtained from short reference speech (seconds) and is used to condition the synthesizer at each time step.
• Training uses a transfer-learning setup where the speaker encoder is frozen when training the synthesizer, with text-to-spectrogram targets derived from phoneme sequences.
• A combined L2 (spectrogram) and L1 loss is used to train the synthesizer targets to improve robustness to noisy data.
• Evaluation includes subjective MOS for naturalness and similarity, and objective speaker-verification EER on unseen speakers.

Experimental results

Research questions

• RQ1 Can a separately trained speaker encoder learned from large-scale, untranscribed data enable zero-shot multispeaker TTS for unseen speakers?
• RQ2 How does the diversity and size of the speaker encoder training data affect naturalness and speaker similarity of synthesized speech?
• RQ3 To what extent can the system transfer speaker characteristics to voices not seen during training, including across datasets with different noise levels and accents?

Key findings

• The proposed model achieves approximately 4.0 MOS across VCTK and LibriSpeech for seen and unseen speakers.
• On LibriSpeech, naturalness for unseen speakers reaches 4.12 MOS, close to seen-speech performance, while VCTK shows stronger similarity due to cleaner data.
• Speaker similarity degrades for unseen speakers, with lower MOS when testing on LibriSpeech; similarity improves when the synthesizer is trained with more diverse speaker data (e.g., LibriSpeech plus VoxCeleb).
• Using a large, diverse, pre-trained speaker encoder (trained on thousands of speakers) substantially improves zero-shot transfer quality and enables generation from fictitious speakers that are distinct from training data.
• Cross-dataset evaluation indicates that LibriSpeech-trained synthesizers generalize better to unseen data than VCTK-trained ones, highlighting the importance of diverse training sources.
• Speaker verification EERs show that LibriSpeech-trained synthesizers produce voices more similar to real speakers than VCTK-trained ones, with synthetic voices generally not confusable with real voices.