[Paper Review] ESPnet2-TTS: Extending the Edge of TTS Research

Tomoki Hayashi, Ryuichi Yamamoto+8 more|arXiv (Cornell University)|Oct 15, 2021

Topic Modeling47 references28 citations

TL;DR

ESPnet2-TTS is an end-to-end TTS toolkit that adds on-the-fly preprocessing, a Model Zoo, and state-of-the-art TTS models with joint training to improve performance and reproducibility; experiments on English and Japanese show results comparable to ground-truth across single/multi-speaker settings.

ABSTRACT

This paper describes ESPnet2-TTS, an end-to-end text-to-speech (E2E-TTS) toolkit. ESPnet2-TTS extends our earlier version, ESPnet-TTS, by adding many new features, including: on-the-fly flexible pre-processing, joint training with neural vocoders, and state-of-the-art TTS models with extensions like full-band E2E text-to-waveform modeling, which simplify the training pipeline and further enhance TTS performance. The unified design of our recipes enables users to quickly reproduce state-of-the-art E2E-TTS results. We also provide many pre-trained models in a unified Python interface for inference, offering a quick means for users to generate baseline samples and build demos. Experimental evaluations with English and Japanese corpora demonstrate that our provided models synthesize utterances comparable to ground-truth ones, achieving state-of-the-art TTS performance. The toolkit is available online at https://github.com/espnet/espnet.

Motivation & Objective

• Advance end-to-end TTS by providing a flexible, scalable toolkit with unified task design.
• Enable quick reproduction of state-of-the-art TTS results through unified recipes and pre-trained models.
• Demonstrate performance of E2E-T2W models and extensions on English and Japanese corpora.
• Highlight joint training with neural vocoders to simplify pipelines and improve quality.

Proposed method

• Introduce ESPnet2-TTS with unified task design enabling on-the-fly preprocessing and a Model Zoo for quick model access.
• Support both autoregressive and non-autoregressive T2M models (Tacotron 2, Transformer-TTS, FastSpeech, FastSpeech 2, Conformer-FastSpeech variants) and multi-speaker extensions via speaker embeddings, X-vectors, and GSTs.
• Provide M2W vocoders (Griffin-Lim, Parallel WaveGAN, MelGAN, StyleMelGAN, HiFi-GAN and multi-band variants) and Joint-T2W training using random windowed discriminators.
• Incorporate E2E-T2W models like VITS with Conformer encoder, full-band waveform modeling, and zero-shot speaker adaptation using pre-trained speaker embeddings.
• Offer evaluation metrics (MCD, F0 RMSE, CER, MOS) and guidance for web-based MOS testing; release pretrained weights and demos.

Experimental results

Research questions

• RQ1 Can ESPnet2-TTS reproduce state-of-the-art E2E-TTS results across multiple languages and speaker settings?
• RQ2 How does joint training of T2M and M2W components affect naturalness and intelligibility compared to standalone training?
• RQ3 What is the impact of using Conformer architectures, full-band waveform modeling, and zero-shot speaker adaptation on TTS performance?
• RQ4 How do single-speaker, multi-speaker, and adaptation scenarios compare in English and Japanese corpora?
• RQ5 What are the trade-offs between AR and NAR T2M models within the ESPnet2-TTS ecosystem?

Key findings

• New state-of-the-art-like performance on English and Japanese corpora with results comparable to ground-truth in several settings.
• Joint training of T2M and M2W (Joint-T2W) improves naturalness and reduces alignment/mismatch issues, especially for non-autoregressive models.
• Conformer-based extensions (Conformer-FastSpeech, Conformer-FastSpeech 2) enable faster inference while maintaining quality.
• VITS-based E2E-T2W with full-band waveform modeling and zero-shot speaker adaptation achieves competitive MOS and intelligibility; performance depends on G2P accuracy.
• Multi-speaker evaluations show X-vectors can enhance speaker similarity and naturalness, with seen/unseen speaker results favoring X-VITS in many cases.
• Extensive English, Japanese, and adaptation experiments demonstrate reproducibility and broad language coverage.