[Paper Review] Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning

Wei Ping, Kainan Peng+6 more|arXiv (Cornell University)|Oct 20, 2017

Speech Recognition and SynthesisComputer Science264 citations

TL;DR

Deep Voice 3 is a fully convolutional, attention-based TTS system that trains much faster, scales to large multi-speaker datasets, and supports multiple vocoders while mitigating attention errors for production-ready throughput.

ABSTRACT

We present Deep Voice 3, a fully-convolutional attention-based neural text-to-speech (TTS) system. Deep Voice 3 matches state-of-the-art neural speech synthesis systems in naturalness while training ten times faster. We scale Deep Voice 3 to data set sizes unprecedented for TTS, training on more than eight hundred hours of audio from over two thousand speakers. In addition, we identify common error modes of attention-based speech synthesis networks, demonstrate how to mitigate them, and compare several different waveform synthesis methods. We also describe how to scale inference to ten million queries per day on one single-GPU server.

Motivation & Objective

• Develop a fully-convolutional sequence-to-sequence TTS model that enables fast training and scalable multi-speaker synthesis.
• Mitigate common attention failure modes in neural TTS to improve fidelity and reliability.
• Evaluate the impact of different waveform synthesis methods on naturalness and deployment efficiency.
• Demonstrate production-ready inference throughput on a single-GPU server for high-traffic deployment.

Proposed method

• Propose a fully-convolutional encoder–decoder–converter architecture for text-to-spectrogram conversion and vocoder parameter prediction.
• Use a multi-hop, convolutional attention mechanism with positional encodings to encourage monotonic attention.
• Incorporate joint character and phoneme representations to improve pronunciation and reduce attention errors.
• Train with multi-task losses combining mel-spectrogram prediction (decoder) and vocoder parameter prediction (converter).
• Evaluate across single-speaker and multi-speaker datasets using Griffin-Lim, WORLD, and WaveNet vocoders.

Experimental results

Research questions

• RQ1 Can a fully-convolutional attention-based TTS model achieve state-of-the-art naturalness while significantly speeding up training?
• RQ2 How well does the model scale to large, multi-speaker datasets (e.g., LibriSpeech, VCTK) and diverse accents?
• RQ3 What are effective strategies to mitigate attention errors (repeats, mispronunciations, skips) in production-ready TTS?
• RQ4 How do different waveform synthesis methods impact naturalness and deployment throughput?

Key findings

• Training is an order of magnitude faster than comparable recurrent architectures (e.g., 0.06s per iteration on one GPU vs 0.59s for Tacotron).
• The model scales to LibriSpeech (820 hours, 2484 speakers) and VCTK (44 hours, 108 speakers) with competitive naturalness.
• Attention errors are largely reduced by joint character–phoneme representations and inference-time monotonic attention; several configurations show substantially fewer repeats/mispronunciations/skips.
• MOS on single-speaker data varies by vocoder: Griffin-Lim 3.62, WORLD 3.63, WaveNet 3.78 (higher is better); Tacotron (WaveNet) 3.78; Deep Voice 2 (WaveNet) 2.74.
• Multi-speaker MOS: Griffin-Lim 3.01 (VCTK), WORLD 3.44 (VCTK), Griffin-Lim 2.37 (LibriSpeech), WORLD 2.89 (LibriSpeech); ground truth 4.69 (VCTK), 4.51 (LibriSpeech).
• Throughput targets ten million queries per day on a single-GPU server with custom kernels and parallel WORLD inference.