[Paper Review] Parallel WaveNet conditioned on VAE latent vectors

Jonas Rohnke, Thomas Merritt+5 more|arXiv (Cornell University)|Dec 17, 2020

Speech Recognition and Synthesis22 references4 citations

TL;DR

This paper proposes conditioning a Parallel WaveNet neural vocoder with latent vectors from a pre-trained VAE to improve speech synthesis quality from predicted mel-spectrograms. By using utterance-level VAE latents during training, the model learns to better reconstruct natural speech, achieving statistically significant quality gains without adding inference latency, as the same fixed mean vector can be used at inference time.

ABSTRACT

Recently the state-of-the-art text-to-speech synthesis systems have shifted to a two-model approach: a sequence-to-sequence model to predict a representation of speech (typically mel-spectrograms), followed by a 'neural vocoder' model which produces the time-domain speech waveform from this intermediate speech representation. This approach is capable of synthesizing speech that is confusable with natural speech recordings. However, the inference speed of neural vocoder approaches represents a major obstacle for deploying this technology for commercial applications. Parallel WaveNet is one approach which has been developed to address this issue, trading off some synthesis quality for significantly faster inference speed. In this paper we investigate the use of a sentence-level conditioning vector to improve the signal quality of a Parallel WaveNet neural vocoder. We condition the neural vocoder with the latent vector from a pre-trained VAE component of a Tacotron 2-style sequence-to-sequence model. With this, we are able to significantly improve the quality of vocoded speech.

Motivation & Objective

• Improve the signal quality of neural vocoders when synthesizing from predicted mel-spectrograms, which often deviate from natural speech due to prediction errors.
• Address the trade-off between inference speed and audio quality in neural vocoders, particularly for real-time commercial TTS applications.
• Explore whether global conditioning via VAE latent vectors can bridge the gap between predicted and natural spectrograms during vocoding.
• Enable high-quality inference without additional latency by decoupling conditioning vector computation from inference time.

Proposed method

• Condition the Parallel WaveNet neural vocoder with a sentence-level latent vector extracted from a pre-trained VAE encoder in a Tacotron 2-style system.
• Use a teacher-student distillation framework: train a full autoregressive WaveNet as a teacher, then distill its output distribution into a student model using Inverse Autoregressive Flows (IAFs) for parallel inference.
• Train the student model to predict IAF affine parameters conditioned on both the mel-spectrogram and the VAE latent vector, enabling joint modeling of spectrogram and global content representation.
• Extract the VAE latent vector from the reference encoder of the acoustic model during training, using it as an additional conditioning signal to guide the vocoder toward more natural speech waveforms.
• During inference, replace dynamic VAE vector computation with a fixed mean vector from the training distribution, maintaining high quality without added latency.
• Use PCA visualization and MUSHRA testing to evaluate the impact of different latent vectors on synthesis quality.

Experimental results

Research questions

• RQ1 Can conditioning a Parallel WaveNet vocoder with a VAE latent vector significantly improve speech quality when synthesizing from predicted mel-spectrograms?
• RQ2 Does the choice of latent vector at inference time affect the quality, or can a fixed representative vector (e.g., mean of training distribution) be used without loss of performance?
• RQ3 What happens when the latent vector is far outside the training distribution—does the model become unstable?
• RQ4 Can this conditioning method be applied to universal vocoders that generalize across speakers and styles?
• RQ5 Is pre-training of the VAE component necessary, or could the global representation be jointly learned with the vocoder?

Key findings

• Conditioning the Parallel WaveNet vocoder with VAE latent vectors led to a statistically significant improvement in MUSHRA scores, with gains of 1 to 1.5 points over the baseline.
• The p-value for the t-test between the baseline and VAE-conditioned systems was less than 0.05, confirming the improvement was statistically significant.
• There was no significant difference in MUSHRA scores between systems conditioned on different latent vectors (p > 0.05), indicating that the specific vector used at inference is not critical as long as it lies within the training distribution.
• When conditioned with a latent vector far outside the training distribution, the model produced only noise, demonstrating that the model is sensitive to out-of-distribution inputs.
• The model achieved improved synthesis quality without adding inference latency, as the fixed mean vector from the training distribution could be used instead of computing dynamic VAE vectors at inference time.
• The results suggest that the VAE latent vector acts as a global prior that helps the vocoder better reconstruct natural speech, especially for expressive voices, by aligning the model’s output with the natural speech space.