[Paper Review] End-to-End Adversarial Text-to-Speech

Jeff Donahue, Sander Dieleman+3 more|arXiv (Cornell University)|Jun 5, 2020

Music and Audio Processing75 references33 citations

TL;DR

The paper presents EATS, an end-to-end text-to-speech system that maps characters or phonemes directly to raw audio using a differentiable aligner and adversarial training, achieving MOS near state-of-the-art with minimal intermediate supervision.

ABSTRACT

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.

Motivation & Objective

• Motivate and develop an end-to-end TTS model that operates directly on character or phoneme inputs to produce raw audio.
• Eliminate traditional multi-stage TTS bottlenecks by learning intermediate representations within a single differentiable generator.
• Leverage adversarial feedback and spectrogram-based prediction losses to guide alignment and audio fidelity.
• Enable efficient feed-forward training and sampling suitable for real-time or accelerated inference.
• Demonstrate that end-to-end learning can approach state-of-the-art quality with reduced supervision.

Proposed method

• A two-block generator comprising a differentiable aligner (200 Hz aligned representations via monotonic interpolation) and a decoder (upsampling to 24 kHz) to produce raw audio.
• An input-conditioned, speaker-embedded latent space used by a GAN-TTS-inspired decoder to generate waveforms.
• An ensemble of random window discriminators (RWDs) and a spectrogram discriminator to provide adversarial feedback on raw audio and its spectral properties.
• A spectrogram prediction loss in the log-mel domain to bootstrap training and guide alignment, complemented by a soft dynamic time warping (soft-DTW) loss to relax exact alignment.
• A monotonic interpolation-based aligner, with token length prediction guiding temporal alignment and a length loss to encourage realistic utterance durations.
• Text pre-processing via phoneme or character input, with optional phonemisation and normalisation, plus padding to accommodate silence.

Experimental results

Research questions

• RQ1 Can an end-to-end feed-forward TTS model synthesize natural-sounding speech directly from text or phonemes without intermediate aligned linguistic features?
• RQ2 How can differentiable alignment and adversarial feedback enable accurate duration modelling and waveform generation in an end-to-end TTS framework?
• RQ3 Does incorporating soft dynamic time warping and spectrogram-based losses improve alignment learning and audio fidelity in a non-autoregressive generator?
• RQ4 What is the impact of multi-speaker training data versus single-speaker data on naturalness (MOS) in an end-to-end TTS system?
• RQ5 How close can end-to-end adversarial TTS get to state-of-the-art quality with reduced supervision?

Key findings

• The base multi-speaker EATS model achieves a mean opinion score (MOS) of 4.083 ± 0.049, approaching state-of-the-art results.
• Ablations show that removing random window discriminators, mel-spectrogram discriminators, or discriminators in general degrades MOS, highlighting the importance of adversarial feedback.
• The DTW-based spectrogram prediction loss and the monotonic interpolation aligner substantially improve temporal fidelity and naturalness compared to alternatives.
• Training on a large multi-speaker dataset (69 speakers, ~260 hours) improves MOS over single-speaker training (MOS 3.829 for the single-speaker ablation).
• Phoneme-based input improves pronunciation reliability over raw character input, yielding higher MOS in the reported evaluations.
• The model runs in feed-forward mode and can generate speech at 200x real-time on a single NVIDIA V100 GPU.