[Paper Review] Deep Voice: Real-time Neural Text-to-Speech

Sercan Ö. Arık, Mike Chrzanowski+10 more|arXiv (Cornell University)|Feb 25, 2017

Speech Recognition and Synthesis20 references395 citations

TL;DR

This paper presents Deep Voice, a fully neural, production-quality TTS system with five components (G2P, segmentation, duration, F0, and audio synthesis) and demonstrates faster-than-real-time, optimized WaveNet inference and training on large datasets.

ABSTRACT

We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks. Deep Voice lays the groundwork for truly end-to-end neural speech synthesis. The system comprises five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For the segmentation model, we propose a novel way of performing phoneme boundary detection with deep neural networks using connectionist temporal classification (CTC) loss. For the audio synthesis model, we implement a variant of WaveNet that requires fewer parameters and trains faster than the original. By using a neural network for each component, our system is simpler and more flexible than traditional text-to-speech systems, where each component requires laborious feature engineering and extensive domain expertise. Finally, we show that inference with our system can be performed faster than real time and describe optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x speedups over existing implementations.

Motivation & Objective

• Replace traditional hand-engineered TTS pipelines with a fully neural end-to-end system.
• Demonstrate real-time capable inference through optimized WaveNet-based audio synthesis.
• Show adaptability to new datasets with minimal manual annotation and hyperparameter tuning.
• Develop efficient training and inference pipelines across CPU and GPU hardware.

Proposed method

• Five-block TTS architecture: grapheme-to-phoneme, segmentation (CTC-based), phoneme duration, F0, and audio synthesis."
• Grapheme-to-phoneme model based on a multi-layer bidirectional encoder with GRU decoder and beam search."
• Segmentation model uses convolutional recurrent network with CTC loss and phoneme-pair labeling to improve boundary detection."
• Joint phoneme duration and F0 model predicting duration, voicing, and 20 time-dependent F0 values with a combined loss."
• Audio synthesis model is a variant of WaveNet using a QRNN pre-encoder to improve training speed and a real-time capable, optimized inference pipeline; CPU and GPU kernels described for real-time performance.

Experimental results

Research questions

• RQ1 Can a production-quality TTS system be built entirely from neural components without hand-engineered features?
• RQ2 Is real-time or faster-than-real-time audio synthesis achievable with optimized WaveNet-based vocoders?
• RQ3 How do the segmentation, duration, and F0 prediction components affect overall speech naturalness and intelligibility?
• RQ4 How does the system perform across different datasets (internal English, Blizzard Blizzard 2013) and what are the perceptual outcomes?
• RQ5 What are the trade-offs between model size, inference speed, and audio quality in a production setting?

Key findings

• Phoneme boundary detection via CTC-based segmentation yields a 7% phoneme-pair error rate after ~14k iterations.
• Grapheme-to-phoneme model attains phoneme error rate of 5.8% and word error rate of 28.7% on CMUDict-style data without a language model.
• Mean absolute error for phoneme duration is 38 ms and F0 error is 29.4 Hz after ~20k iterations.
• 40-layer WaveNet with QRNN pre-encoder produces high-quality speech; 20/30/40 layer models yield usable audio, with 40 layers offering less noise."
• MOS results show ground-truth 48 kHz speech around 4.75±0.12; synthesized 40-layer model with 48 kHz companded audio around 3.84±0.24; synthesized with ground-truth durations/F0 around 2.00±0.23; real-time CPU/GPU inference benchmarks demonstrate real-time and near real-time performance depending on model size.