[Paper Review] Voice Imitating Text-to-Speech Neural Networks

Younggun Lee, Taesu Kim+1 more|arXiv (Cornell University)|Jun 4, 2018

Speech Recognition and Synthesis11 references3 citations

TL;DR

This paper proposes a neural text-to-speech model that imitates a new speaker's voice using only a 6-second speech sample, without requiring transcripts or additional training. By combining a speaker embedder network with Tacotron, the model generates high-quality, speaker-specific speech in real time, achieving voice quality comparable to multi-speaker TTS models while enabling instant voice imitation across diverse inputs.

ABSTRACT

We propose a neural text-to-speech (TTS) model that can imitate a new speaker's voice using only a small amount of speech sample. We demonstrate voice imitation using only a 6-seconds long speech sample without any other information such as transcripts. Our model also enables voice imitation instantly without additional training of the model. We implemented the voice imitating TTS model by combining a speaker embedder network with a state-of-the-art TTS model, Tacotron. The speaker embedder network takes a new speaker's speech sample and returns a speaker embedding. The speaker embedding with a target sentence are fed to Tacotron, and speech is generated with the new speaker's voice. We show that the speaker embeddings extracted by the speaker embedder network can represent the latent structure in different voices. The generated speech samples from our model have comparable voice quality to the ones from existing multi-speaker TTS models.

Motivation & Objective

• To enable instant voice imitation of a new speaker using only a short (6-second) speech sample, without requiring transcripts or additional training.
• To develop a neural TTS system that can generate natural-sounding speech with a target speaker's voice characteristics using a single, brief audio clip.
• To explore the versatility of speaker embedding extraction by allowing the speaker embedder to process various input modalities beyond speech.
• To evaluate the perceptual quality and speaker discriminability of generated speech compared to standard multi-speaker TTS models.

Proposed method

• A speaker embedder network is trained to extract a compact speaker embedding from a short audio clip of a new speaker.
• The speaker embedding is concatenated with the text input and fed into a modified Tacotron model to generate speech conditioned on both the text and the speaker's voice.
• The Tacotron-based TTS model is fine-tuned end-to-end with the speaker embedder to enable joint optimization of voice cloning and speech generation.
• Principal component analysis (PCA) is used to visualize the latent structure of learned speaker embeddings, assessing their ability to capture voice characteristics.
• Crowdsourced surveys on Amazon Mechanical Turk are conducted to quantitatively compare voice quality and speaker discriminability between the proposed model and a baseline multi-speaker TTS model.
• The speaker embedder is designed to generalize across input sources, enabling potential extension to multimodal inputs such as video for emotion-aware voice synthesis.

Experimental results

Research questions

• RQ1 Can a neural TTS model imitate a new speaker’s voice using only a 6-second audio sample without any transcript or additional training?
• RQ2 How does the voice quality of the proposed voice imitating TTS compare to that of a standard multi-speaker TTS model?
• RQ3 To what extent can the generated speech preserve distinct speaker identity, especially when trained on minimal data?
• RQ4 Can the speaker embedder network generalize to other input modalities, such as video, to enable multimodal voice synthesis?

Key findings

• The proposed voice imitating TTS model achieved a mean score of -0.10 ± 1.16 on a crowdsourced comparison survey, indicating no significant perceptual difference in voice quality compared to the multi-speaker TTS baseline.
• The speaker identification accuracy for the voice imitating TTS was 60.1%, significantly above the 50% random baseline, demonstrating that generated voices retain distinguishable speaker characteristics.
• The multi-speaker TTS model achieved a higher speaker identification accuracy of 70.5%, reflecting the advantage of training on larger, diverse datasets compared to the zero-shot imitation approach.
• The speaker embeddings extracted by the speaker embedder network effectively represent the latent structure of different voices, as confirmed by PCA visualization.
• The model enables instant voice imitation without retraining, making it suitable for real-time applications with minimal input.
• The approach is extensible to multimodal inputs, such as facial videos, suggesting potential for emotion- and context-aware text-to-speech synthesis.