[Paper Review] High Fidelity Speech Synthesis with Adversarial Networks

Mikołaj Bińkowski, Jeff Donahue+6 more|arXiv (Cornell University)|Sep 25, 2019

Speech and Audio Processing52 references104 citations

TL;DR

GAN-TTS uses a feed-forward generator and an ensemble of random-window discriminators to synthesize high-fidelity raw audio for text-to-speech, achieving MOS comparable to WaveNet with efficient parallel generation. It also introduces conditional and unconditional DeepSpeech-based metrics for evaluation.

ABSTRACT

Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech. Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes. The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced. To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator. Listen to GAN-TTS reading this abstract at https://storage.googleapis.com/deepmind-media/research/abstract.wav.

Motivation & Objective

• Demonstrate that adversarially trained feed-forward networks can generate high-fidelity raw speech waveform.
• Propose an ensemble of random window discriminators (conditional and unconditional) to assess realism and text-utterance alignment.
• Introduce objective speech-generation metrics based on Fréchet and Kernel distances using DeepSpeech features.
• Evaluate GAN-TTS against autoregressive baselines and perform ablations to validate architectural choices.

Proposed method

• Propose GAN-TTS: a conditional feed-forward generator producing 24 kHz raw audio from 200 Hz linguistic/pitch features.
• Implement an ensemble of Random Window Discriminators (RWDs) at multiple window sizes, with both conditional and unconditional variants.
• Train with adversarial losses using the RWD ensemble to encourage realism and text-utterance consistency.
• Evaluate with subjective MOS and objective metrics: FDSD/KDSD and cFDSD/cKDSD based on DeepSpeech features.
• Use mu-law encoding and dilated conv blocks with residual connections in the generator for long-range dependencies.

Experimental results

Research questions

• RQ1 Can a non-autoregressive, feed-forward generator paired with an ensemble of discriminators produce speech of comparable naturalness to autoregressive models?
• RQ2 Do random-window discriminators across multiple window sizes improve realism and text-utterance alignment?
• RQ3 Are DeepSpeech-based Fréchet and Kernel distances reliable correlates of human MOS for TTS models?
• RQ4 How do various configurations of conditional versus unconditional discriminators impact quality and evaluation metrics?

Key findings

• The best GAN-TTS model achieves MOS of 4.213±0.046, comparable to strong baselines like WaveNet.
• Full multi-window discriminator ensembles outperform single discriminators and deterministic full discriminators in MOS and metrics.
• Unconditional RWDs improve performance; combining multiple conditional RWDs with unconditional RWDs yields best results among ablations.
• Conditional/Unconditional Fréchet DeepSpeech Distance (FDSD) and Kernel DeepSpeech Distance (KDSD) correlate with MOS, supporting their usefulness for evaluation.
• GAN-TTS offers comparable naturalness to autoregressive models while enabling more parallelizable, efficient waveform generation.