[论文解读] End-to-End Adversarial Text-to-Speech

Jeff Donahue, Sander Dieleman+3 more|arXiv (Cornell University)|Jun 5, 2020

Music and Audio Processing参考文献 75被引用 33

一句话总结

论文提出 EATS，一种端到端文本到语音系统，使用可微分对齐器和对抗训练将字符或音素直接映射到原始音频，在最少中间监督下实现接近现有技术水平的 MOS。

ABSTRACT

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.

研究动机与目标

• Motivate and develop an end-to-end TTS model that operates directly on character or phoneme inputs to produce raw audio.
• Eliminate traditional multi-stage TTS bottlenecks by learning intermediate representations within a single differentiable generator.
• Leverage adversarial feedback and spectrogram-based prediction losses to guide alignment and audio fidelity.
• Enable efficient feed-forward training and sampling suitable for real-time or accelerated inference.
• Demonstrate that end-to-end learning can approach state-of-the-art quality with reduced supervision.

提出的方法

• A two-block generator comprising a differentiable aligner (200 Hz aligned representations via monotonic interpolation) and a decoder (upsampling to 24 kHz) to produce raw audio.
• An input-conditioned, speaker-embedded latent space used by a GAN-TTS-inspired decoder to generate waveforms.
• An ensemble of random window discriminators (RWDs) and a spectrogram discriminator to provide adversarial feedback on raw audio and its spectral properties.
• A spectrogram prediction loss in the log-mel domain to bootstrap training and guide alignment, complemented by a soft dynamic time warping (soft-DTW) loss to relax exact alignment.
• A monotonic interpolation-based aligner, with token length prediction guiding temporal alignment and a length loss to encourage realistic utterance durations.
• Text pre-processing via phoneme or character input, with optional phonemisation and normalisation, plus padding to accommodate silence.

实验结果

研究问题

• RQ1 Can an end-to-end feed-forward TTS model synthesize natural-sounding speech directly from text or phonemes without intermediate aligned linguistic features?
• RQ2 How can differentiable alignment and adversarial feedback enable accurate duration modelling and waveform generation in an end-to-end TTS framework?
• RQ3 Does incorporating soft dynamic time warping and spectrogram-based losses improve alignment learning and audio fidelity in a non-autoregressive generator?
• RQ4 What is the impact of multi-speaker training data versus single-speaker data on naturalness (MOS) in an end-to-end TTS system?
• RQ5 How close can end-to-end adversarial TTS get to state-of-the-art quality with reduced supervision?

主要发现

• The base multi-speaker EATS model achieves a mean opinion score (MOS) of 4.083 ± 0.049, approaching state-of-the-art results.
• Ablations show that removing random window discriminators, mel-spectrogram discriminators, or discriminators in general degrades MOS, highlighting the importance of adversarial feedback.
• The DTW-based spectrogram prediction loss and the monotonic interpolation aligner substantially improve temporal fidelity and naturalness compared to alternatives.
• Training on a large multi-speaker dataset (69 speakers, ~260 hours) improves MOS over single-speaker training (MOS 3.829 for the single-speaker ablation).
• Phoneme-based input improves pronunciation reliability over raw character input, yielding higher MOS in the reported evaluations.
• The model runs in feed-forward mode and can generate speech at 200x real-time on a single NVIDIA V100 GPU.