[Paper Review] Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis

Eric Battenberg, Soroosh Mariooryad+5 more|arXiv (Cornell University)|Jun 8, 2019

Speech Recognition and Synthesis24 references43 citations

TL;DR

Capacitron introduces embedding capacity as a unifying framework for variational end-to-end TTS, showing that conditioning the variational posterior on text and speaker enables versatile transfer and high-quality prior sampling, with hierarchical latents controlling fidelity and variation.

ABSTRACT

Recent work has explored sequence-to-sequence latent variable models for expressive speech synthesis (supporting control and transfer of prosody and style), but has not presented a coherent framework for understanding the trade-offs between the competing methods. In this paper, we propose embedding capacity (the amount of information the embedding contains about the data) as a unified method of analyzing the behavior of latent variable models of speech, comparing existing heuristic (non-variational) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information. In our proposed model (Capacitron), we show that by adding conditional dependencies to the variational posterior such that it matches the form of the true posterior, the same model can be used for high-precision prosody transfer, text-agnostic style transfer, and generation of natural-sounding prior samples. For multi-speaker models, Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior. Lastly, we introduce a method for decomposing embedding capacity hierarchically across two sets of latents, allowing a portion of the latent variability to be specified and the remaining variability sampled from a learned prior. Audio examples are available on the web.

Motivation & Objective

• Motivate embedding capacity (representational mutual information) as a unified lens to analyze latent-variable TTS models.
• Show that the KL term upper-bounds embedding capacity and can be controlled with a Lagrange multiplier to target a desired capacity.
• Demonstrate that conditioning the variational posterior on text and speaker enables multi-task transfer (prosody transfer, style transfer) and natural prior samples.
• Introduce hierarchical latent structure to decouple high- and low-level variability and enable controlled transfer and sampling.

Proposed method

• Define embedding capacity as the mutual-information-like quantity bounded by KL(q(z|x)||p(z)).
• Use a Lagrangian objective min_theta max_beta>=0 with a capacity constraint C to bound average embedding capacity (R^avg).
• Match the variational posterior to the true posterior by injecting text and speaker information into the posterior predictor.
• Adopt a hierarchical latent scheme (z_H, z_L) with capacity caps C_H and C_L to separate high- and low-level variability.
• Evaluate by comparing same-text transfer, inter-text style transfer, inter-speaker transfer, and prior sampling across models with and without conditional dependencies.
• Use 128-dimensional embeddings as the fixed capacity setting in experiments.

Experimental results

Research questions

• RQ1 Can embedding capacity be used as a unified measure to compare variational and heuristic TTS methods for prosody/style transfer?
• RQ2 Does conditioning the variational posterior on text and speaker enable high-fidelity transfer and natural prior sampling across tasks?
• RQ3 How does hierarchical latent decomposition affect transfer fidelity and sample diversity?
• RQ4 Can a fixed capacity bound via a Lagrangian approach yield robust transfer and sampling across single- and multi-speaker TTS?

Key findings

• Embedding capacity controls enable consistent transfer behavior: higher capacity with text and speaker conditioning yields better transfer without sacrificing prior-sample quality.
• Var+Txt+Spk achieves stronger inter-speaker transfer and preserves target speaker identity closer to ground truth than variants without speaker conditioning (SpkID and MOS improvements).
• Sampling via the prior remains viable and natural-sounding when conditional dependencies are included, and hierarchical latents allow trading off fidelity vs. sample variation (through C_H and C_L).
• In inter-speaker transfer, Var+Txt+Spk reaches MOS 4.099 and SpkID 95.8% on transferred samples, with prior-sample MOS 3.906 and SpkID 94.9%; the baseline shows MOS ~4.086 and SpkID ~95.7% (ground truth ~4.535 MOS, 96.9% SpkID).
• Hierarchical latents show that increasing C_H reduces reference distance (MCD-DTW) while increasing C_L boosts sample-to-sample variation, enabling controlled transfer realism.