[Paper Review] StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

Yinghao Aaron Li, Cong Han+3 more|PubMed|Jun 13, 2023

Speech Recognition and Synthesis61 references23 citations

TL;DR

StyleTTS 2 introduces style diffusion and adversarial training with large speech language models to achieve human-level TTS, surpassing single-speaker human recordings on LJSpeech and matching multispeaker human performance on VCTK, with strong zero-shot speaker adaptation on LibriTTS.

ABSTRACT

In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github.io/.

Motivation & Objective

• Model human-level TTS by modeling style as a latent variable via diffusion, eliminating the need for reference speech during generation.
• Leverage large pre-trained speech language models as discriminators to improve naturalness through adversarial training.
• Enable end-to-end training with differentiable duration modeling to improve stability and synthesis quality.
• Demonstrate strong performance on single-speaker and multispeaker datasets, and show data-efficient zero-shot speaker adaptation on LibriTTS.

Proposed method

• Model speech style as a latent variable sampled by a diffusion model conditioned on input text.
• Use end-to-end training to directly generate waveforms from text, style, and prosody without fixed vocoders.
• Replace mel-spectrogram generation with a waveform decoder (HifiGAN or iSTFTNet) and apply AdaIN for style conditioning.
• Incorporate large SLMs (e.g., WavLM) as discriminators with a differentiable duration model to enable SLM-based adversarial training (L_slm).
• Employ differentiable duration modeling to map predicted phoneme durations to frameupsampling in a differentiable way (Gaussian upsampling and a non-parametric approach).
• Handle multispeaker setting by conditioning diffusion on speaker reference embeddings and using adaptive styling for speaker adaptation.

Experimental results

Research questions

• RQ1 Can style diffusion enable diverse, high-quality TTS without reference speech while maintaining efficiency?
• RQ2 Do large pre-trained SLMs as discriminators improve naturalness and robustness in adversarially trained TTS?
• RQ3 Does end-to-end training with differentiable duration modeling yield human-level naturalness and speaker similarity on standard datasets?
• RQ4 How does StyleTTS 2 perform in single-speaker vs. multispeaker settings, and in zero-shot speaker adaptation?
• RQ5 Is StyleTTS 2 robust to out-of-distribution texts and data-efficient with limited training data?

Key findings

• StyleTTS 2 achieves CMOS of +0.28 over ground truth on LJSpeech (p<0.05) and CMOS of +1.07 over NaturalSpeech (p<<0.01).
• On multispeaker VCTK, StyleTTS 2 achieves CMOS of −0.02 for naturalness and +0.30 for similarity (p>0.05 and p<0.1 respectively) compared to reference.
• StyleTTS 2 attains MOS of 3.83 on LJSpeech, surpassing previous models, and matches human-level naturalness on VCTK (CMOS close to ground truth).
• In zero-shot LibriTTS adaptation, StyleTTS 2 outperforms Vall-E in naturalness with CMOS +0.67 (p<0.01) while using ~250x less training data (245 hours vs 60k hours).
• StyleTTS 2 demonstrates strong OOD robustness, outperforming baselines in MOS-N for OOD texts and showing minimal degradation in naturalness for unseen content.
• The approach enables end-to-end differentiable training with style diffusion and SLM-based adversarial losses, achieving human-level TTS on public single- and multispeaker datasets.