[Paper Review] Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation

Dongchan Min, Dong Bok Lee+2 more|arXiv (Cornell University)|Jun 6, 2021

Speech Recognition and Synthesis45 citations

TL;DR

Meta-StyleSpeech extends StyleSpeech with meta-learning and discriminators to achieve strong one-shot, short-reference speaker adaptation for multi-speaker TTS. It delivers high-quality, speaker-consistent synthesis even from <1 second reference audio.

ABSTRACT

With rapid progress in neural text-to-speech (TTS) models, personalized speech generation is now in high demand for many applications. For practical applicability, a TTS model should generate high-quality speech with only a few audio samples from the given speaker, that are also short in length. However, existing methods either require to fine-tune the model or achieve low adaptation quality without fine-tuning. In this work, we propose StyleSpeech, a new TTS model which not only synthesizes high-quality speech but also effectively adapts to new speakers. Specifically, we propose Style-Adaptive Layer Normalization (SALN) which aligns gain and bias of the text input according to the style extracted from a reference speech audio. With SALN, our model effectively synthesizes speech in the style of the target speaker even from single speech audio. Furthermore, to enhance StyleSpeech's adaptation to speech from new speakers, we extend it to Meta-StyleSpeech by introducing two discriminators trained with style prototypes, and performing episodic training. The experimental results show that our models generate high-quality speech which accurately follows the speaker's voice with single short-duration (1-3 sec) speech audio, significantly outperforming baselines.

Motivation & Objective

• Motivate high-quality, expressive multi-speaker TTS that can adapt to a new speaker with only a few short audio samples.
• Introduce a style-adaptive normalization mechanism to condition synthesis on a reference speaker’s voice and prosody.
• Enhance adaptation to unseen speakers through meta-learning with adversarial discriminators and style prototypes.
• Demonstrate state-of-the-art performance on seen and unseen speaker tasks, including ultra-short reference audio scenarios.

Proposed method

• Propose StyleSpeech with Style-Adaptive Layer Normalization (SALN) to align gain and bias of features based on a style vector extracted from a reference speech.
• Extend StyleSpeech to Meta-StyleSpeech by adding a style discriminator with style prototypes and a phoneme discriminator, enabling episodic meta-learning for one-shot adaptation.
• Use a mel-style encoder to extract a style vector from a reference speech and feed SALN into the FFT-based phoneme encoder and mel-spectrogram decoder of a FastSpeech2–based generator.
• Train with reconstruction loss on seen speakers and adversarial losses (LS-GAN) from two discriminators for unseen-speaker adaptation, plus a style prototype classification objective.
• Perform episodic training by simulating support/query splits per episode to improve generalization to unseen speakers.

Experimental results

Research questions

• RQ1 Can StyleSpeech synthesize high-quality multi-speaker speech and adapt to new speakers using a single short reference audio?
• RQ2 Does meta-learning with discriminators improve adaptation quality for unseen speakers compared to non-meta-trained baselines?
• RQ3 How does the length of the reference speech affect adaptation accuracy and voice similarity for unseen speakers?
• RQ4 What is the contribution of the style and phoneme discriminators and style prototypes to unseen-speaker adaptation?
• RQ5 How close can generated speech be to the target speaker's voice in objective and subjective metrics across seen and unseen conditions?

Key findings

• StyleSpeech achieves higher MOS and lower MCD/WER than baselines for seen speakers.
• Meta-StyleSpeech attains the best performance among evaluated models for seen speakers.
• For unseen speakers, Meta-StyleSpeech outperforms baselines in MOS, MCD, WER, and voice similarity.
• Adaptation with reference audio shorter than 1 second is substantially improved by Meta-StyleSpeech, outperforming StyleSpeech across all reference lengths.
• Ablation studies show the importance of the phoneme discriminator, style discriminator, and style prototypes for unseen-speaker adaptation.