[Paper Review] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Chengyi Wang, Sanyuan Chen+11 more|arXiv (Cornell University)|Jan 5, 2023

Speech Recognition and Synthesis161 citations

TL;DR

The paper presents VALL-E, a language-model-based TTS system that uses discrete audio codec codes and prompts from enrolled recordings to perform zero-shot TTS with strong naturalness and speaker similarity, trained on 60k hours of data.

ABSTRACT

We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.

Motivation & Objective

• Scale TTS training data to hundreds of thousands of hours to improve zero-shot generalization.
• Treat TTS as conditional language modeling using discrete audio codec codes as intermediate representations.
• Enable prompt-based, in-context learning to synthesize unseen speakers from a 3-second enrolled recording.
• Preserve speaker emotion and acoustic environment during synthesis.
• Demonstrate superior zero-shot performance against state-of-the-art baselines on LibriSpeech and VCTK.

Proposed method

• Represent speech with discrete codes from an off-the-shelf neural audio codec (EnCodec).
• Form TTS as conditional codec language modeling to generate the acoustic code matrix given phoneme prompts and acoustic prompts from an enrolled recording.
• Use a hierarchical model with an autoregressive decoder for the first quantizer and non-autoregressive models for subsequent quantizers to balance quality and speed.
• Train on 60k hours of LibriLight data with ~7k speakers, using ASR-generated transcripts for supervision.
• Employ prompt-based inference with phoneme prompts and acoustic prompts to enable zero-shot synthesis of unseen speakers.
• Evaluate with WER and speaker similarity metrics, plus human CMOS/SMOS ratings.

Figure 1: The overview of VALL-E . Unlike the previous pipeline (e.g., phoneme $\rightarrow$ mel-spectrogram $\rightarrow$ waveform), the pipeline of VALL-E is phoneme $\rightarrow$ discrete code $\rightarrow$ waveform. VALL-E generates the discrete audio codec codes based on phoneme and acoustic co

Experimental results

Research questions

• RQ1 Can zero-shot TTS be achieved by framing synthesis as conditional language modeling over discrete acoustic codes?
• RQ2 Does scaling up semi-supervised speech data improve zero-shot TTS performance and generalization to unseen speakers?
• RQ3 Can prompt-based, in-context learning enable natural and speaker-faithful speech without fine-tuning?
• RQ4 How do AR and NAR components contribute to synthesis quality and inference speed?
• RQ5 What is the robustness of VALL-E across standard TTS and cross-dataset evaluations (LibriSpeech, VCTK)?

Key findings

• VALL-E significantly outperforms the state-of-the-art zero-shot TTS baseline on LibriSpeech and VCTK in terms of speech naturalness and speaker similarity.
• On LibriSpeech, VALL-E achieves CMOS improvements of +0.12 and SMOS improvements of +0.93 over the baseline; on VCTK, CMOS +0.23 and SMOS +0.04 over ground truth.
• VALL-E-continual (using ground-truth 3s prompt for continuation) reduces WER to 3.8 and maintains strong speaker similarity (SMOS 0.508).
• Human evaluation shows VALL-E close to ground truth on SMOS (0. difference within margin) and CMOS improvements over baseline (+0.12 on LibriSpeech; +0.23 on VCTK).
• Ablation studies show crucial roles for phoneme prompts (reducing WER) and acoustic prompts (boosting speaker similarity).
• Compared to other speech-to-speech LM systems (GSLM, AudioLM), VALL-E demonstrates higher robustness and speaker fidelity in zero-shot TTS.

Figure 2: The neural audio codec model revisit. Because RVQ is employed, the first quantizer plays the most important role in reconstruction, and the impact from others gradually decreases.