[Paper Review] Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech

Popov Va, Ivan Vovk+3 more|arXiv (Cornell University)|May 13, 2021

Music and Audio Processing31 references43 citations

TL;DR

Grad-TTS introduces a diffusion-based acoustic feature generator for TTS, using a score-based decoder and Monotonic Alignment Search to produce mel-spectrograms; it offers a controllable speed-quality trade-off and competitive MOS scores.

ABSTRACT

Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.

Motivation & Objective

• Motivate diffusion probabilistic modelling for acoustic feature generation in TTS.
• Develop Grad-TTS to transform Gaussian noise into mel-spectrograms conditioned on encoded text via MAS-aligned features.
• Enable end-to-end TTS potential by replacing mel-spectrogram output with raw waveform generation in an extended setup.
• Provide a mechanism to trade off inference speed and quality by varying the number of diffusion steps.

Proposed method

• Formulate forward diffusion that maps data to Gaussian noise with a generalized mean/diagonal covariance (mu, Sigma).
• Derive reverse diffusion via an SDE/ODE with score-based gradient estimation using a neural network s_theta(X_t, mu, t).
• Train with a score-matching-like loss to estimate gradients of the noisy data distribution, incorporating an encoder-based alignment mu.
• Use Monotonic Alignment Search (MAS) to obtain text-to-aligned-mel-spectrogram mappings and train a duration predictor.
• Adopt a U-Net decoder conditioned on the encoder output, operating on multi-resolution mel-spectrogram representations and enabling variable inference steps.

Experimental results

Research questions

• RQ1 Can a diffusion probabilistic model serve as an effective acoustic feature generator for TTS?
• RQ2 Does MAS-aligned encoding plus a score-based decoder yield competitive speech quality with adjustable inference speed?
• RQ3 How does Grad-TTS compare to autoregressive and non-autoregressive baselines in MOS and objective metrics?
• RQ4 Is it feasible to extend Grad-TTS to end-to-end TTS by generating waveform directly from diffusion decoding?
• RQ5 What are the effects of varying the number of diffusion steps on speech quality and real-time performance?

Key findings

• Grad-TTS with diffusion-based decoding achieves MOS competitive with Tacotron2 and close to ground-truth under certain settings.
• Increasing the number of reverse-diffusion iterations improves MOS, but gains saturate beyond around 10–100 iterations depending on settings.
• Grad-TTS-1000 attains MOS near ground truth (4.44±0.05 vs 4.53±0.06) with real-time capable GPU synthesis when using a sufficiently high iteration count.
• The model demonstrates real-time synthesis on GPU for decoder steps below 100, with roughly 15M parameters, and outperforms some baselines in speed and quality trade-offs.
• End-to-end diffusion-based TTS is feasible but currently underperforms the Mel-spectrogram based Grad-TTS in quality, suggesting future potential for WaveGrad-like end-to-end diffusion setups.
• Compared to Glow-TTS and FastSpeech, Grad-TTS shows competitive MOS and improved reliability in alignment and pronunciation.