[논문 리뷰] Deep Voice: Real-time Neural Text-to-Speech

Sercan Ö. Arık, Mike Chrzanowski외 10명|arXiv (Cornell University)|2017. 02. 25.

Speech Recognition and Synthesis참고 문헌 20인용 수 395

한 줄 요약

이 논문은 다섯 구성요소(G2P, 세그먼테이션, 지속시간, F0, 그리고 오디오 합성)를 갖춘 완전한 신경망 생산 품질 TTS 시스템인 Deep Voice를 제시하고 대규모 데이터 세트에서 WaveNet 추론과 학습을 실시간보다 빠르게 최적화하는 것을 시연합니다.

ABSTRACT

We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks. Deep Voice lays the groundwork for truly end-to-end neural speech synthesis. The system comprises five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For the segmentation model, we propose a novel way of performing phoneme boundary detection with deep neural networks using connectionist temporal classification (CTC) loss. For the audio synthesis model, we implement a variant of WaveNet that requires fewer parameters and trains faster than the original. By using a neural network for each component, our system is simpler and more flexible than traditional text-to-speech systems, where each component requires laborious feature engineering and extensive domain expertise. Finally, we show that inference with our system can be performed faster than real time and describe optimized WaveNet inference kernels on both CPU and GPU that achieve up to 400x speedups over existing implementations.

연구 동기 및 목표

• 전통적인 수작업으로 엔지니어링된 TTS 파이프라인을 완전한 신경망 엔드-투-엔드 시스템으로 대체한다.
• 최적화된 WaveNet 기반 오디오 합성을 통해 실시간 가능한 추론을 시연한다.
• 수동 주석 및 하이퍼파라미터 튜닝을 최소화하고 새로운 데이터 세트에 대한 적응성을 보여준다.
• CPU 및 GPU 하드웨어에서 효율적인 학습 및 추론 파이프라인을 개발한다.

제안 방법

• Five-block TTS architecture: grapheme-to-phoneme, segmentation (CTC-based), phoneme duration, F0, and audio synthesis."
• Grapheme-to-phoneme model based on a multi-layer bidirectional encoder with GRU decoder and beam search."
• Segmentation model uses convolutional recurrent network with CTC loss and phoneme-pair labeling to improve boundary detection."
• Joint phoneme duration and F0 model predicting duration, voicing, and 20 time-dependent F0 values with a combined loss."
• Audio synthesis model is a variant of WaveNet using a QRNN pre-encoder to improve training speed and a real-time capable, optimized inference pipeline; CPU and GPU kernels described for real-time performance.

실험 결과

연구 질문

• RQ1 Can a production-quality TTS system be built entirely from neural components without hand-engineered features?
• RQ2 Is real-time or faster-than-real-time audio synthesis achievable with optimized WaveNet-based vocoders?
• RQ3 How do the segmentation, duration, and F0 prediction components affect overall speech naturalness and intelligibility?
• RQ4 How does the system perform across different datasets (internal English, Blizzard Blizzard 2013) and what are the perceptual outcomes?
• RQ5 What are the trade-offs between model size, inference speed, and audio quality in a production setting?

주요 결과

• Phoneme boundary detection via CTC-based segmentation yields a 7% phoneme-pair error rate after ~14k iterations.
• Grapheme-to-phoneme model attains phoneme error rate of 5.8% and word error rate of 28.7% on CMUDict-style data without a language model.
• Mean absolute error for phoneme duration is 38 ms and F0 error is 29.4 Hz after ~20k iterations.
• 40-layer WaveNet with QRNN pre-encoder produces high-quality speech; 20/30/40 layer models yield usable audio, with 40 layers offering less noise."
• MOS results show ground-truth 48 kHz speech around 4.75±0.12; synthesized 40-layer model with 48 kHz companded audio around 3.84±0.24; synthesized with ground-truth durations/F0 around 2.00±0.23; real-time CPU/GPU inference benchmarks demonstrate real-time and near real-time performance depending on model size.